> We tried building recently released Spark 0.8.0 with latest version of > hadoop-client (2.1.0-beta) and YARN support. According to release notes, > 0.8.0 should support YARN. The build, however, fails with compilation error. > Command line used was SPARK_HADOOP_VERSION=2.1.0-beta SPARK_YARN=true > sbt/sbt assembly, the log file is attached. Latest master code also fails. > Building without YARN or with previous (2.0.6-alpha) version of > hadoop-client works. >
1. There are some incompatible changes to yarn release 2.1.0-beta compared to yarn release 2.0.6 alpha. You can read about the changes at the links provided below. Some of the changes are YARN-439 Flatten NodeHeartbeatResponse, YARN-440 Flatten RegisterNodeManagerResponse by Xuan Gong via sseth. a. http://hadoop.apache.org/docs/r2.1.0-beta/hadoop-project-dist/hadoop-yarn/CHANGES.txt b. http://hadoop.apache.org/docs/r2.1.0-beta/hadoop-project-dist/hadoop-common/releasenotes.html c. http://hortonworks.com/blog/stabilizing-yarn-apis-for-apache-hadoop-2-beta-and-beyond/#.UkNaC7yoUkA 2. Currently, you can build spark with the earlier 2.0.6-alpha release. When using spark release 0.7.3 or release 0.8.0 you can build for use with the YARN resource allocator by issuing the following command from the install directory. SPARK_HADOOP_VERSION=2.0.5-alpha SPARK_YARN=true sbt/sbt assembly http://spark.incubator.apache.org/docs/latest/running-on-yarn.html > Are we doing something wrong or is the support for YARN on 2.1.0-beta not > yet there? We would prefer not to downgrade from Hadoop 2 beta to alpha. > > -- > Best regards, > Sergey Parhomenko
