Are you setting -Pyarn-alpha? ./sbt/sbt -Pyarn-alpha, followed by
"projects", shows it as a module. You should only build yarn-stable
*or* yarn-alpha at any given time.
I don't remember the modules changing in a while. 'yarn-alpha' is for
YARN before it stabilized, circa early Hadoop 2.0.x. 'yarn-stable' is
for beta and stable YARN, circa late Hadoop 2.0.x and onwards. 'yarn'
is code common to both, so should compile with yarn-alpha.
What's the compile error, and are you setting yarn.version? the
default is to use hadoop.version, but that defaults to 1.0.4 and there
is no such YARN.
Unless I missed it, I only see compile errors in yarn-stable, and you
are trying to compile vs YARN alpha versions no?
On Thu, Jul 17, 2014 at 5:39 AM, Chester Chen <ches...@alpinenow.com> wrote:
> Looking further, the yarn and yarn-stable are both for the stable version
> of Yarn, that explains the compilation errors when using 2.0.5-alpha
> version of hadoop.
>
> the module yarn-alpha ( although is still on SparkBuild.scala), is no
> longer there in sbt console.
>
>
>> projects
>
> [info] In file:/Users/chester/projects/spark/
>
> [info] assembly
>
> [info] bagel
>
> [info] catalyst
>
> [info] core
>
> [info] examples
>
> [info] graphx
>
> [info] hive
>
> [info] mllib
>
> [info] oldDeps
>
> [info] repl
>
> [info] spark
>
> [info] sql
>
> [info] streaming
>
> [info] streaming-flume
>
> [info] streaming-kafka
>
> [info] streaming-mqtt
>
> [info] streaming-twitter
>
> [info] streaming-zeromq
>
> [info] tools
>
> [info] yarn
>
> [info] * yarn-stable
>
>
> On Wed, Jul 16, 2014 at 5:41 PM, Chester Chen <ches...@alpinenow.com> wrote:
>
>> Hmm
>> looks like a Build script issue:
>>
>> I run the command with :
>>
>> sbt/sbt clean *yarn/*test:compile
>>
>> but errors came from
>>
>> [error] 40 errors found
>>
>> [error] (*yarn-stable*/compile:compile) Compilation failed
>>
>>
>> Chester
>>
>>
>> On Wed, Jul 16, 2014 at 5:18 PM, Chester Chen <ches...@alpinenow.com>
>> wrote:
>>
>>> Hi, Sandy
>>>
>>> We do have some issue with this. The difference is in Yarn-Alpha and
>>> Yarn Stable ( I noticed that in the latest build, the module name has
>>> changed,
>>> yarn-alpha --> yarn
>>> yarn --> yarn-stable
>>> )
>>>
>>> For example: MRJobConfig.class
>>> the field:
>>> "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH"
>>>
>>>
>>> In Yarn-Alpha : the field returns java.lang.String[]
>>>
>>> java.lang.String[] DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
>>>
>>> while in Yarn-Stable, it returns a String
>>>
>>> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
>>>
>>> So in ClientBaseSuite.scala
>>>
>>> The following code:
>>>
>>> val knownDefMRAppCP: Seq[String] =
>>> getFieldValue[*String*, Seq[String]](classOf[MRJobConfig],
>>>
>>> "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH",
>>> Seq[String]())(a =>
>>> *a.split(",")*)
>>>
>>>
>>> works for yarn-stable, but doesn't work for yarn-alpha.
>>>
>>> This is the only failure for the SNAPSHOT I downloaded 2 weeks ago. I
>>> believe this can be refactored to yarn-alpha module and make different
>>> tests according different API signatures.
>>>
>>> I just update the master branch and build doesn't even compile for
>>> Yarn-Alpha (yarn) model. Yarn-Stable compile with no error and test passed.
>>>
>>>
>>> Does the Spark Jenkins job run against yarn-alpha ?
>>>
>>>
>>>
>>>
>>>
>>> Here is output from yarn-alpha compilation:
>>>
>>> I got the 40 compilation errors.
>>>
>>> sbt/sbt clean yarn/test:compile
>>>
>>> Using /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home as
>>> default JAVA_HOME.
>>>
>>> Note, this will be overridden by -java-home if it is set.
>>>
>>> [info] Loading project definition from
>>> /Users/chester/projects/spark/project/project
>>>
>>> [info] Loading project definition from
>>> /Users/chester/.sbt/0.13/staging/ec3aa8f39111944cc5f2/sbt-pom-reader/project
>>>
>>> [warn] Multiple resolvers having different access mechanism configured
>>> with same name 'sbt-plugin-releases'. To avoid conflict, Remove duplicate
>>> project resolvers (`resolvers`) or rename publishing resolver (`publishTo`).
>>>
>>> [info] Loading project definition from
>>> /Users/chester/projects/spark/project
>>>
>>> NOTE: SPARK_HADOOP_VERSION is deprecated, please use
>>> -Dhadoop.version=2.0.5-alpha
>>>
>>> NOTE: SPARK_YARN is deprecated, please use -Pyarn flag.
>>>
>>> [info] Set current project to spark-parent (in build
>>> file:/Users/chester/projects/spark/)
>>>
>>> [success] Total time: 0 s, completed Jul 16, 2014 5:13:06 PM
>>>
>>> [info] Updating {file:/Users/chester/projects/spark/}core...
>>>
>>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
>>>
>>> [info] Done updating.
>>>
>>> [info] Updating {file:/Users/chester/projects/spark/}yarn...
>>>
>>> [info] Updating {file:/Users/chester/projects/spark/}yarn-stable...
>>>
>>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
>>>
>>> [info] Done updating.
>>>
>>> [info] Resolving commons-net#commons-net;3.1 ...
>>>
>>> [info] Compiling 358 Scala sources and 34 Java sources to
>>> /Users/chester/projects/spark/core/target/scala-2.10/classes...
>>>
>>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
>>>
>>> [info] Done updating.
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/hadoop/mapred/SparkHadoopMapRedUtil.scala:43:
>>> constructor TaskAttemptID in class TaskAttemptID is deprecated: see
>>> corresponding Javadoc for more information.
>>>
>>> [warn] new TaskAttemptID(jtIdentifier, jobId, isMap, taskId,
>>> attemptId)
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:501:
>>> constructor Job in class Job is deprecated: see corresponding Javadoc for
>>> more information.
>>>
>>> [warn] val job = new NewHadoopJob(hadoopConfiguration)
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:634:
>>> constructor Job in class Job is deprecated: see corresponding Javadoc for
>>> more information.
>>>
>>> [warn] val job = new NewHadoopJob(conf)
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:167:
>>> constructor TaskID in class TaskID is deprecated: see corresponding Javadoc
>>> for more information.
>>>
>>> [warn] new TaskAttemptID(new TaskID(jID.value, true, splitID),
>>> attemptID))
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:188:
>>> method makeQualified in class Path is deprecated: see corresponding Javadoc
>>> for more information.
>>>
>>> [warn] outputPath.makeQualified(fs)
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:84:
>>> method isDir in class FileStatus is deprecated: see corresponding Javadoc
>>> for more information.
>>>
>>> [warn] if (!fs.getFileStatus(path).isDir) {
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:118:
>>> method isDir in class FileStatus is deprecated: see corresponding Javadoc
>>> for more information.
>>>
>>> [warn] val logDirs = if (logStatus != null)
>>> logStatus.filter(_.isDir).toSeq else Seq[FileStatus]()
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/input/WholeTextFileInputFormat.scala:56:
>>> method isDir in class FileStatus is deprecated: see corresponding Javadoc
>>> for more information.
>>>
>>> [warn] if (file.isDir) 0L else file.getLen
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/CheckpointRDD.scala:110:
>>> method getDefaultReplication in class FileSystem is deprecated: see
>>> corresponding Javadoc for more information.
>>>
>>> [warn] fs.create(tempOutputPath, false, bufferSize,
>>> fs.getDefaultReplication, blockSize)
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:267:
>>> constructor TaskID in class TaskID is deprecated: see corresponding Javadoc
>>> for more information.
>>>
>>> [warn] val taId = new TaskAttemptID(new TaskID(jobID, true,
>>> splitId), attemptId)
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:767:
>>> constructor Job in class Job is deprecated: see corresponding Javadoc for
>>> more information.
>>>
>>> [warn] val job = new NewAPIHadoopJob(hadoopConf)
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:830:
>>> constructor Job in class Job is deprecated: see corresponding Javadoc for
>>> more information.
>>>
>>> [warn] val job = new NewAPIHadoopJob(hadoopConf)
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:185:
>>> method isDir in class FileStatus is deprecated: see corresponding Javadoc
>>> for more information.
>>>
>>> [warn] fileStatuses.filter(!_.isDir).map(_.getPath).toSeq
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/InputFormatInfo.scala:106:
>>> constructor Job in class Job is deprecated: see corresponding Javadoc for
>>> more information.
>>>
>>> [warn] val job = new Job(conf)
>>>
>>> [warn] ^
>>>
>>> [warn] 14 warnings found
>>>
>>> [warn] Note:
>>> /Users/chester/projects/spark/core/src/main/java/org/apache/spark/api/java/JavaSparkContextVarargsWorkaround.java
>>> uses unchecked or unsafe operations.
>>>
>>> [warn] Note: Recompile with -Xlint:unchecked for details.
>>>
>>> [info] Compiling 15 Scala sources to
>>> /Users/chester/projects/spark/yarn/stable/target/scala-2.10/classes...
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:26:
>>> object api is not a member of package org.apache.hadoop.yarn.client
>>>
>>> [error] import org.apache.hadoop.yarn.client.api.YarnClient
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:40:
>>> not found: value YarnClient
>>>
>>> [error] val yarnClient = YarnClient.createYarnClient
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:32:
>>> object api is not a member of package org.apache.hadoop.yarn.client
>>>
>>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:33:
>>> object api is not a member of package org.apache.hadoop.yarn.client
>>>
>>> [error] import
>>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:36:
>>> object util is not a member of package org.apache.hadoop.yarn.webapp
>>>
>>> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:63:
>>> value RM_AM_MAX_ATTEMPTS is not a member of object
>>> org.apache.hadoop.yarn.conf.YarnConfiguration
>>>
>>> [error] YarnConfiguration.RM_AM_MAX_ATTEMPTS,
>>> YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS)
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:65:
>>> not found: type AMRMClient
>>>
>>> [error] private var amClient: AMRMClient[ContainerRequest] = _
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:91:
>>> not found: value AMRMClient
>>>
>>> [error] amClient = AMRMClient.createAMRMClient()
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:136:
>>> not found: value WebAppUtils
>>>
>>> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:40:
>>> object api is not a member of package org.apache.hadoop.yarn.client
>>>
>>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:618:
>>> not found: type AMRMClient
>>>
>>> [error] amClient: AMRMClient[ContainerRequest],
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:596:
>>> not found: type AMRMClient
>>>
>>> [error] amClient: AMRMClient[ContainerRequest],
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:577:
>>> not found: type AMRMClient
>>>
>>> [error] amClient: AMRMClient[ContainerRequest],
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:452:
>>> value CONTAINER_ID is not a member of object
>>> org.apache.hadoop.yarn.api.ApplicationConstants.Environment
>>>
>>> [error] val containerIdString = System.getenv(
>>> ApplicationConstants.Environment.CONTAINER_ID.name())
>>>
>>> [error]
>>> ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:128:
>>> value setTokens is not a member of
>>> org.apache.hadoop.yarn.api.records.ContainerLaunchContext
>>>
>>> [error] amContainer.setTokens(ByteBuffer.wrap(dob.getData()))
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:36:
>>> object api is not a member of package org.apache.hadoop.yarn.client
>>>
>>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:37:
>>> object api is not a member of package org.apache.hadoop.yarn.client
>>>
>>> [error] import
>>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:39:
>>> object util is not a member of package org.apache.hadoop.yarn.webapp
>>>
>>> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:62:
>>> not found: type AMRMClient
>>>
>>> [error] private var amClient: AMRMClient[ContainerRequest] = _
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:99:
>>> not found: value AMRMClient
>>>
>>> [error] amClient = AMRMClient.createAMRMClient()
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:158:
>>> not found: value WebAppUtils
>>>
>>> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:31:
>>> object ProtoUtils is not a member of package
>>> org.apache.hadoop.yarn.api.records.impl.pb
>>>
>>> [error] import org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:33:
>>> object api is not a member of package org.apache.hadoop.yarn.client
>>>
>>> [error] import org.apache.hadoop.yarn.client.api.NMClient
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:53:
>>> not found: type NMClient
>>>
>>> [error] var nmClient: NMClient = _
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:59:
>>> not found: value NMClient
>>>
>>> [error] nmClient = NMClient.createNMClient()
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:79:
>>> value setTokens is not a member of
>>> org.apache.hadoop.yarn.api.records.ContainerLaunchContext
>>>
>>> [error] ctx.setTokens(ByteBuffer.wrap(dob.getData()))
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:35:
>>> object ApplicationMasterProtocol is not a member of package
>>> org.apache.hadoop.yarn.api
>>>
>>> [error] import org.apache.hadoop.yarn.api.ApplicationMasterProtocol
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:41:
>>> object api is not a member of package org.apache.hadoop.yarn.client
>>>
>>> [error] import
>>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:65:
>>> not found: type AMRMClient
>>>
>>> [error] val amClient: AMRMClient[ContainerRequest],
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:389:
>>> not found: type ContainerRequest
>>>
>>> [error] ): ArrayBuffer[ContainerRequest] = {
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:388:
>>> not found: type ContainerRequest
>>>
>>> [error] hostContainers: ArrayBuffer[ContainerRequest]
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:405:
>>> not found: type ContainerRequest
>>>
>>> [error] val requestedContainers = new
>>> ArrayBuffer[ContainerRequest](rackToCounts.size)
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:434:
>>> not found: type ContainerRequest
>>>
>>> [error] val containerRequests: List[ContainerRequest] =
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:508:
>>> not found: type ContainerRequest
>>>
>>> [error] ): ArrayBuffer[ContainerRequest] = {
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:446:
>>> not found: type ContainerRequest
>>>
>>> [error] val hostContainerRequests = new
>>> ArrayBuffer[ContainerRequest](preferredHostToCount.size)
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:458:
>>> not found: type ContainerRequest
>>>
>>> [error] val rackContainerRequests: List[ContainerRequest] =
>>> createRackResourceRequests(
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:467:
>>> not found: type ContainerRequest
>>>
>>> [error] val containerRequestBuffer = new
>>> ArrayBuffer[ContainerRequest](
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:542:
>>> not found: type ContainerRequest
>>>
>>> [error] ): ArrayBuffer[ContainerRequest] = {
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:545:
>>> value newInstance is not a member of object
>>> org.apache.hadoop.yarn.api.records.Resource
>>>
>>> [error] val resource = Resource.newInstance(memoryRequest,
>>> executorCores)
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:550:
>>> not found: type ContainerRequest
>>>
>>> [error] val requests = new ArrayBuffer[ContainerRequest]()
>>>
>>> [error] ^
>>>
>>> [error] 40 errors found
>>>
>>> [error] (yarn-stable/compile:compile) Compilation failed
>>>
>>> [error] Total time: 98 s, completed Jul 16, 2014 5:14:44 PM
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Jul 16, 2014 at 4:19 PM, Sandy Ryza <sandy.r...@cloudera.com>
>>> wrote:
>>>
>>>> Hi Ron,
>>>>
>>>> I just checked and this bug is fixed in recent releases of Spark.
>>>>
>>>> -Sandy
>>>>
>>>>
>>>> On Sun, Jul 13, 2014 at 8:15 PM, Chester Chen <ches...@alpinenow.com>
>>>> wrote:
>>>>
>>>>> Ron,
>>>>> Which distribution and Version of Hadoop are you using ?
>>>>>
>>>>> I just looked at CDH5 ( hadoop-mapreduce-client-core-
>>>>> 2.3.0-cdh5.0.0),
>>>>>
>>>>> MRJobConfig does have the field :
>>>>>
>>>>> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
>>>>>
>>>>> Chester
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Jul 13, 2014 at 6:49 PM, Ron Gonzalez <zlgonza...@yahoo.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>> I was doing programmatic submission of Spark yarn jobs and I saw
>>>>>> code in ClientBase.getDefaultYarnApplicationClasspath():
>>>>>>
>>>>>> val field =
>>>>>> classOf[MRJobConfig].getField("DEFAULT_YARN_APPLICATION_CLASSPATH)
>>>>>> MRJobConfig doesn't have this field so the created launch env is
>>>>>> incomplete. Workaround is to set yarn.application.classpath with the
>>>>>> value
>>>>>> from YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH.
>>>>>>
>>>>>> This results in having the spark job hang if the submission config is
>>>>>> different from the default config. For example, if my resource manager
>>>>>> port
>>>>>> is 8050 instead of 8030, then the spark app is not able to register
>>>>>> itself
>>>>>> and stays in ACCEPTED state.
>>>>>>
>>>>>> I can easily fix this by changing this to YarnConfiguration instead of
>>>>>> MRJobConfig but was wondering what the steps are for submitting a fix.
>>>>>>
>>>>>> Thanks,
>>>>>> Ron
>>>>>>
>>>>>> Sent from my iPhone
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
