Hi Sergey, Because this was a breaking API change on YARN's part, I'd recommend just sticking with 2.0.x for now if possible. Otherwise, we'll likely add support for this, and remove support for older versions of YARN, in the next major version of Spark. Before that, it's possible that we can make a branch for this if there's demand for it.
Matei On Sep 27, 2013, at 5:12 PM, Sergey Parhomenko <[email protected]> wrote: > Hi Stoney, > > Thanks for info. A couple of follow-up questions: > - If I build Spark 0.8.0 with hadoop-client 2.0.5-alpha, can I use it safely > to connect to Hadoop (HDFS) server running 2.1.0-beta? > - Do you know if there plans / outlook on making the master compatible with > 2.1.0-beta? > > -- > Best regards, > Sergey Parhomenko > > > On 27 September 2013 20:25, Stoney Vintson <[email protected]> wrote: > > We tried building recently released Spark 0.8.0 with latest version of > > hadoop-client (2.1.0-beta) and YARN support. According to release notes, > > 0.8.0 should support YARN. The build, however, fails with compilation error. > > Command line used was SPARK_HADOOP_VERSION=2.1.0-beta SPARK_YARN=true > > sbt/sbt assembly, the log file is attached. Latest master code also fails. > > Building without YARN or with previous (2.0.6-alpha) version of > > hadoop-client works. > > > > 1. There are some incompatible changes to yarn release 2.1.0-beta > compared to yarn release 2.0.6 alpha. You can read about the changes > at the links provided below. Some of the changes are YARN-439 Flatten > NodeHeartbeatResponse, YARN-440 Flatten RegisterNodeManagerResponse by > Xuan Gong via sseth. > > a. > http://hadoop.apache.org/docs/r2.1.0-beta/hadoop-project-dist/hadoop-yarn/CHANGES.txt > b. > http://hadoop.apache.org/docs/r2.1.0-beta/hadoop-project-dist/hadoop-common/releasenotes.html > c. > http://hortonworks.com/blog/stabilizing-yarn-apis-for-apache-hadoop-2-beta-and-beyond/#.UkNaC7yoUkA > > 2. Currently, you can build spark with the earlier 2.0.6-alpha release. > When using spark release 0.7.3 or release 0.8.0 you can build for use > with the YARN resource allocator by issuing the following command from > the install directory. > > SPARK_HADOOP_VERSION=2.0.5-alpha SPARK_YARN=true sbt/sbt assembly > > http://spark.incubator.apache.org/docs/latest/running-on-yarn.html > > > Are we doing something wrong or is the support for YARN on 2.1.0-beta not > > yet there? We would prefer not to downgrade from Hadoop 2 beta to alpha. > > > > -- > > Best regards, > > Sergey Parhomenko >
