Yup, unfortunately YARN changed its API upon releasing 2.2, which puts us in an awkward position because all the major current users are on the old YARN API (from 0.23.x and 2.0.x) but new users will try this one. We'll probably change the default version in Spark 0.8.1 or 0.8.2. If you look on this list, there were some earlier messages on how to patch Spark to compile against the new YARN.
Matei On Oct 23, 2013, at 11:39 PM, Pei-Lun Lee <[email protected]> wrote: > Hi, > > YARN has already reached stable release 2.2.0. But due to API change Spark > cannot build. > > Here is the build log: > > root@master:~/spark# SPARK_HADOOP_VERSION=2.2.0 SPARK_YARN=true ./sbt/sbt > assembly > [info] Loading project definition from /root/spark/project/project > [info] Loading project definition from /root/spark/project > [info] Set current project to root (in build file:/root/spark/) > [info] Compiling 8 Scala sources to > /root/spark/yarn/target/scala-2.9.3/classes... > [error] > /root/spark/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:42: > not found: type AMRMProtocol > [error] private var resourceManager: AMRMProtocol = null > [error] ^ > [error] > /root/spark/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:126: > not found: type AMRMProtocol > [error] private def registerWithResourceManager(): AMRMProtocol = { > [error] ^ > [error] > /root/spark/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:119: > value AM_CONTAINER_ID_ENV is not a member of object > org.apache.hadoop.yarn.api.ApplicationConstants > [error] val containerIdString = > envs.get(ApplicationConstants.AM_CONTAINER_ID_ENV) > [error] ^ > [error] > /root/spark/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:131: > not found: type AMRMProtocol > [error] return rpc.getProxy(classOf[AMRMProtocol], rmAddress, > conf).asInstanceOf[AMRMProtocol] > [error] ^ > [error] > /root/spark/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:131: > not found: type AMRMProtocol > [error] return rpc.getProxy(classOf[AMRMProtocol], rmAddress, > conf).asInstanceOf[AMRMProtocol] > [error] > ^ > [error] > /root/spark/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:138: > value setApplicationAttemptId is not a member of > org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterRequest > [error] appMasterRequest.setApplicationAttemptId(appAttemptId) > [error] ^ > > ... (CUT) > > Is there plan to support YARN 2.2.0? > > Cheers, > -- > Pei-Lun Lee
