To add, we've made some effort to yarn-alpha to work with the 2.0.x line,
but this was a time when YARN went through wild API changes. The only line
that the yarn-alpha profile is guaranteed to work against is the 0.23 line.
On Thu, Jul 17, 2014 at 12:40 AM, Sean Owen <so...@cloudera.com> wrote:
> Are you setting -Pyarn-alpha? ./sbt/sbt -Pyarn-alpha, followed by
> "projects", shows it as a module. You should only build yarn-stable
> *or* yarn-alpha at any given time.
>
> I don't remember the modules changing in a while. 'yarn-alpha' is for
> YARN before it stabilized, circa early Hadoop 2.0.x. 'yarn-stable' is
> for beta and stable YARN, circa late Hadoop 2.0.x and onwards. 'yarn'
> is code common to both, so should compile with yarn-alpha.
>
> What's the compile error, and are you setting yarn.version? the
> default is to use hadoop.version, but that defaults to 1.0.4 and there
> is no such YARN.
>
> Unless I missed it, I only see compile errors in yarn-stable, and you
> are trying to compile vs YARN alpha versions no?
>
> On Thu, Jul 17, 2014 at 5:39 AM, Chester Chen <ches...@alpinenow.com>
> wrote:
> > Looking further, the yarn and yarn-stable are both for the stable version
> > of Yarn, that explains the compilation errors when using 2.0.5-alpha
> > version of hadoop.
> >
> > the module yarn-alpha ( although is still on SparkBuild.scala), is no
> > longer there in sbt console.
> >
> >
> >> projects
> >
> > [info] In file:/Users/chester/projects/spark/
> >
> > [info] assembly
> >
> > [info] bagel
> >
> > [info] catalyst
> >
> > [info] core
> >
> > [info] examples
> >
> > [info] graphx
> >
> > [info] hive
> >
> > [info] mllib
> >
> > [info] oldDeps
> >
> > [info] repl
> >
> > [info] spark
> >
> > [info] sql
> >
> > [info] streaming
> >
> > [info] streaming-flume
> >
> > [info] streaming-kafka
> >
> > [info] streaming-mqtt
> >
> > [info] streaming-twitter
> >
> > [info] streaming-zeromq
> >
> > [info] tools
> >
> > [info] yarn
> >
> > [info] * yarn-stable
> >
> >
> > On Wed, Jul 16, 2014 at 5:41 PM, Chester Chen <ches...@alpinenow.com>
> wrote:
> >
> >> Hmm
> >> looks like a Build script issue:
> >>
> >> I run the command with :
> >>
> >> sbt/sbt clean *yarn/*test:compile
> >>
> >> but errors came from
> >>
> >> [error] 40 errors found
> >>
> >> [error] (*yarn-stable*/compile:compile) Compilation failed
> >>
> >>
> >> Chester
> >>
> >>
> >> On Wed, Jul 16, 2014 at 5:18 PM, Chester Chen <ches...@alpinenow.com>
> >> wrote:
> >>
> >>> Hi, Sandy
> >>>
> >>> We do have some issue with this. The difference is in Yarn-Alpha
> and
> >>> Yarn Stable ( I noticed that in the latest build, the module name has
> >>> changed,
> >>> yarn-alpha --> yarn
> >>> yarn --> yarn-stable
> >>> )
> >>>
> >>> For example: MRJobConfig.class
> >>> the field:
> >>> "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH"
> >>>
> >>>
> >>> In Yarn-Alpha : the field returns java.lang.String[]
> >>>
> >>> java.lang.String[] DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
> >>>
> >>> while in Yarn-Stable, it returns a String
> >>>
> >>> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
> >>>
> >>> So in ClientBaseSuite.scala
> >>>
> >>> The following code:
> >>>
> >>> val knownDefMRAppCP: Seq[String] =
> >>> getFieldValue[*String*, Seq[String]](classOf[MRJobConfig],
> >>>
> >>> "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH",
> >>> Seq[String]())(a =>
> >>> *a.split(",")*)
> >>>
> >>>
> >>> works for yarn-stable, but doesn't work for yarn-alpha.
> >>>
> >>> This is the only failure for the SNAPSHOT I downloaded 2 weeks ago. I
> >>> believe this can be refactored to yarn-alpha module and make different
> >>> tests according different API signatures.
> >>>
> >>> I just update the master branch and build doesn't even compile for
> >>> Yarn-Alpha (yarn) model. Yarn-Stable compile with no error and test
> passed.
> >>>
> >>>
> >>> Does the Spark Jenkins job run against yarn-alpha ?
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> Here is output from yarn-alpha compilation:
> >>>
> >>> I got the 40 compilation errors.
> >>>
> >>> sbt/sbt clean yarn/test:compile
> >>>
> >>> Using /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home
> as
> >>> default JAVA_HOME.
> >>>
> >>> Note, this will be overridden by -java-home if it is set.
> >>>
> >>> [info] Loading project definition from
> >>> /Users/chester/projects/spark/project/project
> >>>
> >>> [info] Loading project definition from
> >>>
> /Users/chester/.sbt/0.13/staging/ec3aa8f39111944cc5f2/sbt-pom-reader/project
> >>>
> >>> [warn] Multiple resolvers having different access mechanism configured
> >>> with same name 'sbt-plugin-releases'. To avoid conflict, Remove
> duplicate
> >>> project resolvers (`resolvers`) or rename publishing resolver
> (`publishTo`).
> >>>
> >>> [info] Loading project definition from
> >>> /Users/chester/projects/spark/project
> >>>
> >>> NOTE: SPARK_HADOOP_VERSION is deprecated, please use
> >>> -Dhadoop.version=2.0.5-alpha
> >>>
> >>> NOTE: SPARK_YARN is deprecated, please use -Pyarn flag.
> >>>
> >>> [info] Set current project to spark-parent (in build
> >>> file:/Users/chester/projects/spark/)
> >>>
> >>> [success] Total time: 0 s, completed Jul 16, 2014 5:13:06 PM
> >>>
> >>> [info] Updating {file:/Users/chester/projects/spark/}core...
> >>>
> >>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
> >>>
> >>> [info] Done updating.
> >>>
> >>> [info] Updating {file:/Users/chester/projects/spark/}yarn...
> >>>
> >>> [info] Updating {file:/Users/chester/projects/spark/}yarn-stable...
> >>>
> >>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
> >>>
> >>> [info] Done updating.
> >>>
> >>> [info] Resolving commons-net#commons-net;3.1 ...
> >>>
> >>> [info] Compiling 358 Scala sources and 34 Java sources to
> >>> /Users/chester/projects/spark/core/target/scala-2.10/classes...
> >>>
> >>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
> >>>
> >>> [info] Done updating.
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/hadoop/mapred/SparkHadoopMapRedUtil.scala:43:
> >>> constructor TaskAttemptID in class TaskAttemptID is deprecated: see
> >>> corresponding Javadoc for more information.
> >>>
> >>> [warn] new TaskAttemptID(jtIdentifier, jobId, isMap, taskId,
> >>> attemptId)
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:501:
> >>> constructor Job in class Job is deprecated: see corresponding Javadoc
> for
> >>> more information.
> >>>
> >>> [warn] val job = new NewHadoopJob(hadoopConfiguration)
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:634:
> >>> constructor Job in class Job is deprecated: see corresponding Javadoc
> for
> >>> more information.
> >>>
> >>> [warn] val job = new NewHadoopJob(conf)
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:167:
> >>> constructor TaskID in class TaskID is deprecated: see corresponding
> Javadoc
> >>> for more information.
> >>>
> >>> [warn] new TaskAttemptID(new TaskID(jID.value, true, splitID),
> >>> attemptID))
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:188:
> >>> method makeQualified in class Path is deprecated: see corresponding
> Javadoc
> >>> for more information.
> >>>
> >>> [warn] outputPath.makeQualified(fs)
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:84:
> >>> method isDir in class FileStatus is deprecated: see corresponding
> Javadoc
> >>> for more information.
> >>>
> >>> [warn] if (!fs.getFileStatus(path).isDir) {
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:118:
> >>> method isDir in class FileStatus is deprecated: see corresponding
> Javadoc
> >>> for more information.
> >>>
> >>> [warn] val logDirs = if (logStatus != null)
> >>> logStatus.filter(_.isDir).toSeq else Seq[FileStatus]()
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/input/WholeTextFileInputFormat.scala:56:
> >>> method isDir in class FileStatus is deprecated: see corresponding
> Javadoc
> >>> for more information.
> >>>
> >>> [warn] if (file.isDir) 0L else file.getLen
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/CheckpointRDD.scala:110:
> >>> method getDefaultReplication in class FileSystem is deprecated: see
> >>> corresponding Javadoc for more information.
> >>>
> >>> [warn] fs.create(tempOutputPath, false, bufferSize,
> >>> fs.getDefaultReplication, blockSize)
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:267:
> >>> constructor TaskID in class TaskID is deprecated: see corresponding
> Javadoc
> >>> for more information.
> >>>
> >>> [warn] val taId = new TaskAttemptID(new TaskID(jobID, true,
> >>> splitId), attemptId)
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:767:
> >>> constructor Job in class Job is deprecated: see corresponding Javadoc
> for
> >>> more information.
> >>>
> >>> [warn] val job = new NewAPIHadoopJob(hadoopConf)
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:830:
> >>> constructor Job in class Job is deprecated: see corresponding Javadoc
> for
> >>> more information.
> >>>
> >>> [warn] val job = new NewAPIHadoopJob(hadoopConf)
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:185:
> >>> method isDir in class FileStatus is deprecated: see corresponding
> Javadoc
> >>> for more information.
> >>>
> >>> [warn] fileStatuses.filter(!_.isDir).map(_.getPath).toSeq
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/InputFormatInfo.scala:106:
> >>> constructor Job in class Job is deprecated: see corresponding Javadoc
> for
> >>> more information.
> >>>
> >>> [warn] val job = new Job(conf)
> >>>
> >>> [warn] ^
> >>>
> >>> [warn] 14 warnings found
> >>>
> >>> [warn] Note:
> >>>
> /Users/chester/projects/spark/core/src/main/java/org/apache/spark/api/java/JavaSparkContextVarargsWorkaround.java
> >>> uses unchecked or unsafe operations.
> >>>
> >>> [warn] Note: Recompile with -Xlint:unchecked for details.
> >>>
> >>> [info] Compiling 15 Scala sources to
> >>> /Users/chester/projects/spark/yarn/stable/target/scala-2.10/classes...
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:26:
> >>> object api is not a member of package org.apache.hadoop.yarn.client
> >>>
> >>> [error] import org.apache.hadoop.yarn.client.api.YarnClient
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:40:
> >>> not found: value YarnClient
> >>>
> >>> [error] val yarnClient = YarnClient.createYarnClient
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:32:
> >>> object api is not a member of package org.apache.hadoop.yarn.client
> >>>
> >>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:33:
> >>> object api is not a member of package org.apache.hadoop.yarn.client
> >>>
> >>> [error] import
> >>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:36:
> >>> object util is not a member of package org.apache.hadoop.yarn.webapp
> >>>
> >>> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:63:
> >>> value RM_AM_MAX_ATTEMPTS is not a member of object
> >>> org.apache.hadoop.yarn.conf.YarnConfiguration
> >>>
> >>> [error] YarnConfiguration.RM_AM_MAX_ATTEMPTS,
> >>> YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS)
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:65:
> >>> not found: type AMRMClient
> >>>
> >>> [error] private var amClient: AMRMClient[ContainerRequest] = _
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:91:
> >>> not found: value AMRMClient
> >>>
> >>> [error] amClient = AMRMClient.createAMRMClient()
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:136:
> >>> not found: value WebAppUtils
> >>>
> >>> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:40:
> >>> object api is not a member of package org.apache.hadoop.yarn.client
> >>>
> >>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:618:
> >>> not found: type AMRMClient
> >>>
> >>> [error] amClient: AMRMClient[ContainerRequest],
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:596:
> >>> not found: type AMRMClient
> >>>
> >>> [error] amClient: AMRMClient[ContainerRequest],
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:577:
> >>> not found: type AMRMClient
> >>>
> >>> [error] amClient: AMRMClient[ContainerRequest],
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:452:
> >>> value CONTAINER_ID is not a member of object
> >>> org.apache.hadoop.yarn.api.ApplicationConstants.Environment
> >>>
> >>> [error] val containerIdString = System.getenv(
> >>> ApplicationConstants.Environment.CONTAINER_ID.name())
> >>>
> >>> [error]
> >>> ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:128:
> >>> value setTokens is not a member of
> >>> org.apache.hadoop.yarn.api.records.ContainerLaunchContext
> >>>
> >>> [error] amContainer.setTokens(ByteBuffer.wrap(dob.getData()))
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:36:
> >>> object api is not a member of package org.apache.hadoop.yarn.client
> >>>
> >>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:37:
> >>> object api is not a member of package org.apache.hadoop.yarn.client
> >>>
> >>> [error] import
> >>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:39:
> >>> object util is not a member of package org.apache.hadoop.yarn.webapp
> >>>
> >>> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:62:
> >>> not found: type AMRMClient
> >>>
> >>> [error] private var amClient: AMRMClient[ContainerRequest] = _
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:99:
> >>> not found: value AMRMClient
> >>>
> >>> [error] amClient = AMRMClient.createAMRMClient()
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:158:
> >>> not found: value WebAppUtils
> >>>
> >>> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:31:
> >>> object ProtoUtils is not a member of package
> >>> org.apache.hadoop.yarn.api.records.impl.pb
> >>>
> >>> [error] import org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:33:
> >>> object api is not a member of package org.apache.hadoop.yarn.client
> >>>
> >>> [error] import org.apache.hadoop.yarn.client.api.NMClient
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:53:
> >>> not found: type NMClient
> >>>
> >>> [error] var nmClient: NMClient = _
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:59:
> >>> not found: value NMClient
> >>>
> >>> [error] nmClient = NMClient.createNMClient()
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:79:
> >>> value setTokens is not a member of
> >>> org.apache.hadoop.yarn.api.records.ContainerLaunchContext
> >>>
> >>> [error] ctx.setTokens(ByteBuffer.wrap(dob.getData()))
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:35:
> >>> object ApplicationMasterProtocol is not a member of package
> >>> org.apache.hadoop.yarn.api
> >>>
> >>> [error] import org.apache.hadoop.yarn.api.ApplicationMasterProtocol
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:41:
> >>> object api is not a member of package org.apache.hadoop.yarn.client
> >>>
> >>> [error] import
> >>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:65:
> >>> not found: type AMRMClient
> >>>
> >>> [error] val amClient: AMRMClient[ContainerRequest],
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:389:
> >>> not found: type ContainerRequest
> >>>
> >>> [error] ): ArrayBuffer[ContainerRequest] = {
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:388:
> >>> not found: type ContainerRequest
> >>>
> >>> [error] hostContainers: ArrayBuffer[ContainerRequest]
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:405:
> >>> not found: type ContainerRequest
> >>>
> >>> [error] val requestedContainers = new
> >>> ArrayBuffer[ContainerRequest](rackToCounts.size)
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:434:
> >>> not found: type ContainerRequest
> >>>
> >>> [error] val containerRequests: List[ContainerRequest] =
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:508:
> >>> not found: type ContainerRequest
> >>>
> >>> [error] ): ArrayBuffer[ContainerRequest] = {
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:446:
> >>> not found: type ContainerRequest
> >>>
> >>> [error] val hostContainerRequests = new
> >>> ArrayBuffer[ContainerRequest](preferredHostToCount.size)
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:458:
> >>> not found: type ContainerRequest
> >>>
> >>> [error] val rackContainerRequests: List[ContainerRequest] =
> >>> createRackResourceRequests(
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:467:
> >>> not found: type ContainerRequest
> >>>
> >>> [error] val containerRequestBuffer = new
> >>> ArrayBuffer[ContainerRequest](
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:542:
> >>> not found: type ContainerRequest
> >>>
> >>> [error] ): ArrayBuffer[ContainerRequest] = {
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:545:
> >>> value newInstance is not a member of object
> >>> org.apache.hadoop.yarn.api.records.Resource
> >>>
> >>> [error] val resource = Resource.newInstance(memoryRequest,
> >>> executorCores)
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:550:
> >>> not found: type ContainerRequest
> >>>
> >>> [error] val requests = new ArrayBuffer[ContainerRequest]()
> >>>
> >>> [error] ^
> >>>
> >>> [error] 40 errors found
> >>>
> >>> [error] (yarn-stable/compile:compile) Compilation failed
> >>>
> >>> [error] Total time: 98 s, completed Jul 16, 2014 5:14:44 PM
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Wed, Jul 16, 2014 at 4:19 PM, Sandy Ryza <sandy.r...@cloudera.com>
> >>> wrote:
> >>>
> >>>> Hi Ron,
> >>>>
> >>>> I just checked and this bug is fixed in recent releases of Spark.
> >>>>
> >>>> -Sandy
> >>>>
> >>>>
> >>>> On Sun, Jul 13, 2014 at 8:15 PM, Chester Chen <ches...@alpinenow.com>
> >>>> wrote:
> >>>>
> >>>>> Ron,
> >>>>> Which distribution and Version of Hadoop are you using ?
> >>>>>
> >>>>> I just looked at CDH5 ( hadoop-mapreduce-client-core-
> >>>>> 2.3.0-cdh5.0.0),
> >>>>>
> >>>>> MRJobConfig does have the field :
> >>>>>
> >>>>> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
> >>>>>
> >>>>> Chester
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Sun, Jul 13, 2014 at 6:49 PM, Ron Gonzalez <zlgonza...@yahoo.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Hi,
> >>>>>> I was doing programmatic submission of Spark yarn jobs and I saw
> >>>>>> code in ClientBase.getDefaultYarnApplicationClasspath():
> >>>>>>
> >>>>>> val field =
> >>>>>> classOf[MRJobConfig].getField("DEFAULT_YARN_APPLICATION_CLASSPATH)
> >>>>>> MRJobConfig doesn't have this field so the created launch env is
> >>>>>> incomplete. Workaround is to set yarn.application.classpath with
> the value
> >>>>>> from YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH.
> >>>>>>
> >>>>>> This results in having the spark job hang if the submission config
> is
> >>>>>> different from the default config. For example, if my resource
> manager port
> >>>>>> is 8050 instead of 8030, then the spark app is not able to register
> itself
> >>>>>> and stays in ACCEPTED state.
> >>>>>>
> >>>>>> I can easily fix this by changing this to YarnConfiguration instead
> of
> >>>>>> MRJobConfig but was wondering what the steps are for submitting a
> fix.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Ron
> >>>>>>
> >>>>>> Sent from my iPhone
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
>
