[
https://issues.apache.org/jira/browse/SPARK-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200046#comment-14200046
]
Oliver Bye edited comment on SPARK-2815 at 11/6/14 10:11 AM:
-------------------------------------------------------------
This is still and issue for CDH4.6
I can confirm https://github.com/apache/spark/pull/151/files does fix this
issue. Specifically the patch to
yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala
I've applied 2 line change change based on v1.1.0 and made it available here.
{noformat}
git clone https://github.com/olibye/spark.git --branch SPARK-2815
--single-branch
mvn -T 1C -Pyarn-alpha -Dhadoop.version=2.0.0-cdh4.6.0 -DskipTests install
{noformat}
I've not created a pull request, as this appears to be an unsupported branch,
and would just be a repeat.
was (Author: olibye):
This is still and issue for CDH4.6
I can confirm https://github.com/apache/spark/pull/151/files does fix this
issue. Specifically the patch to
yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala
I've applied 2 line change change based on v1.1.0 and made it available here.
git clone https://github.com/olibye/spark.git --branch SPARK-2815
--single-branch
mvn -T 1C -Pyarn-alpha -Dhadoop.version=2.0.0-cdh4.6.0 -DskipTests install
I've not created a pull request, as this appears to be an unsupported branch,
and would just be a repeat.
> Compilation failed upon the hadoop version 2.0.0-cdh4.5.0
> ---------------------------------------------------------
>
> Key: SPARK-2815
> URL: https://issues.apache.org/jira/browse/SPARK-2815
> Project: Spark
> Issue Type: Bug
> Components: Build
> Affects Versions: 1.1.0
> Reporter: pengyanhong
> Assignee: Guoqiang Li
>
> compile fail via SPARK_HADOOP_VERSION=2.0.0-cdh4.5.0 SPARK_YARN=true
> SPARK_HIVE=true sbt/sbt assembly, finally get error message : [error]
> (yarn-stable/compile:compile) Compilation failed, the following is the detail
> error on console:
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:26:
> object api is not a member of package org.apache.hadoop.yarn.client
> [error] import org.apache.hadoop.yarn.client.api.YarnClient
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:40:
> not found: value YarnClient
> [error] val yarnClient = YarnClient.createYarnClient
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:32:
> object api is not a member of package org.apache.hadoop.yarn.client
> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:33:
> object api is not a member of package org.apache.hadoop.yarn.client
> [error] import org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:36:
> object util is not a member of package org.apache.hadoop.yarn.webapp
> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:64:
> value RM_AM_MAX_ATTEMPTS is not a member of object
> org.apache.hadoop.yarn.conf.YarnConfiguration
> [error] YarnConfiguration.RM_AM_MAX_ATTEMPTS,
> YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS)
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:66:
> not found: type AMRMClient
> [error] private var amClient: AMRMClient[ContainerRequest] = _
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:92:
> not found: value AMRMClient
> [error] amClient = AMRMClient.createAMRMClient()
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:137:
> not found: value WebAppUtils
> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:40:
> object api is not a member of package org.apache.hadoop.yarn.client
> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:618:
> not found: type AMRMClient
> [error] amClient: AMRMClient[ContainerRequest],
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:596:
> not found: type AMRMClient
> [error] amClient: AMRMClient[ContainerRequest],
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:577:
> not found: type AMRMClient
> [error] amClient: AMRMClient[ContainerRequest],
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:410:
> value CONTAINER_ID is not a member of object
> org.apache.hadoop.yarn.api.ApplicationConstants.Environment
> [error] val containerIdString =
> System.getenv(ApplicationConstants.Environment.CONTAINER_ID.name())
> [error]
> ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:128:
> value setTokens is not a member of
> org.apache.hadoop.yarn.api.records.ContainerLaunchContext
> [error] amContainer.setTokens(ByteBuffer.wrap(dob.getData()))
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:33:
> object api is not a member of package org.apache.hadoop.yarn.client
> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:34:
> object api is not a member of package org.apache.hadoop.yarn.client
> [error] import org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:36:
> object util is not a member of package org.apache.hadoop.yarn.webapp
> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:61:
> not found: type AMRMClient
> [error] private var amClient: AMRMClient[ContainerRequest] = _
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:102:
> not found: value AMRMClient
> [error] amClient = AMRMClient.createAMRMClient()
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:166:
> not found: value WebAppUtils
> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:31:
> object ProtoUtils is not a member of package
> org.apache.hadoop.yarn.api.records.impl.pb
> [error] import org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:33:
> object api is not a member of package org.apache.hadoop.yarn.client
> [error] import org.apache.hadoop.yarn.client.api.NMClient
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:53:
> not found: type NMClient
> [error] var nmClient: NMClient = _
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:59:
> not found: value NMClient
> [error] nmClient = NMClient.createNMClient()
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:79:
> value setTokens is not a member of
> org.apache.hadoop.yarn.api.records.ContainerLaunchContext
> [error] ctx.setTokens(ByteBuffer.wrap(dob.getData()))
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:35:
> object ApplicationMasterProtocol is not a member of package
> org.apache.hadoop.yarn.api
> [error] import org.apache.hadoop.yarn.api.ApplicationMasterProtocol
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:41:
> object api is not a member of package org.apache.hadoop.yarn.client
> [error] import org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:65:
> not found: type AMRMClient
> [error] val amClient: AMRMClient[ContainerRequest],
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:389:
> not found: type ContainerRequest
> [error] ): ArrayBuffer[ContainerRequest] = {
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:388:
> not found: type ContainerRequest
> [error] hostContainers: ArrayBuffer[ContainerRequest]
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:405:
> not found: type ContainerRequest
> [error] val requestedContainers = new
> ArrayBuffer[ContainerRequest](rackToCounts.size)
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:434:
> not found: type ContainerRequest
> [error] val containerRequests: List[ContainerRequest] =
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:508:
> not found: type ContainerRequest
> [error] ): ArrayBuffer[ContainerRequest] = {
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:446:
> not found: type ContainerRequest
> [error] val hostContainerRequests = new
> ArrayBuffer[ContainerRequest](preferredHostToCount.size)
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:458:
> not found: type ContainerRequest
> [error] val rackContainerRequests: List[ContainerRequest] =
> createRackResourceRequests(
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:467:
> not found: type ContainerRequest
> [error] val containerRequestBuffer = new
> ArrayBuffer[ContainerRequest](
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:542:
> not found: type ContainerRequest
> [error] ): ArrayBuffer[ContainerRequest] = {
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:545:
> value newInstance is not a member of object
> org.apache.hadoop.yarn.api.records.Resource
> [error] val resource = Resource.newInstance(memoryRequest, executorCores)
> [error] ^
> [error]
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:550:
> not found: type ContainerRequest
> [error] val requests = new ArrayBuffer[ContainerRequest]()
> [error] ^
> [error] 40 errors found
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
