Hi Stoney, Thanks for info. A couple of follow-up questions: - If I build Spark 0.8.0 with hadoop-client 2.0.5-alpha, can I use it safely to connect to Hadoop (HDFS) server running 2.1.0-beta? - Do you know if there plans / outlook on making the master compatible with 2.1.0-beta?
-- Best regards, Sergey Parhomenko On 27 September 2013 20:25, Stoney Vintson <[email protected]> wrote: > > We tried building recently released Spark 0.8.0 with latest version of > > hadoop-client (2.1.0-beta) and YARN support. According to release notes, > > 0.8.0 should support YARN. The build, however, fails with compilation > error. > > Command line used was SPARK_HADOOP_VERSION=2.1.0-beta SPARK_YARN=true > > sbt/sbt assembly, the log file is attached. Latest master code also > fails. > > Building without YARN or with previous (2.0.6-alpha) version of > > hadoop-client works. > > > > 1. There are some incompatible changes to yarn release 2.1.0-beta > compared to yarn release 2.0.6 alpha. You can read about the changes > at the links provided below. Some of the changes are YARN-439 Flatten > NodeHeartbeatResponse, YARN-440 Flatten RegisterNodeManagerResponse by > Xuan Gong via sseth. > > a. > http://hadoop.apache.org/docs/r2.1.0-beta/hadoop-project-dist/hadoop-yarn/CHANGES.txt > b. > http://hadoop.apache.org/docs/r2.1.0-beta/hadoop-project-dist/hadoop-common/releasenotes.html > c. > http://hortonworks.com/blog/stabilizing-yarn-apis-for-apache-hadoop-2-beta-and-beyond/#.UkNaC7yoUkA > > 2. Currently, you can build spark with the earlier 2.0.6-alpha release. > When using spark release 0.7.3 or release 0.8.0 you can build for use > with the YARN resource allocator by issuing the following command from > the install directory. > > SPARK_HADOOP_VERSION=2.0.5-alpha SPARK_YARN=true sbt/sbt assembly > > http://spark.incubator.apache.org/docs/latest/running-on-yarn.html > > > Are we doing something wrong or is the support for YARN on 2.1.0-beta not > > yet there? We would prefer not to downgrade from Hadoop 2 beta to alpha. > > > > -- > > Best regards, > > Sergey Parhomenko >
