[
https://issues.apache.org/jira/browse/SPARK-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14340433#comment-14340433
]
Thomas Graves commented on SPARK-6050:
--------------------------------------
Note, the problem is in determining whether cpu scheduling is off or not.
There is no interface exposed to the user. For the Capacity Scheduler you would
have to look for specific config:
capacity-scheduler.xml.
<name>yarn.scheduler.capacity.resource-calculator</name>
<value>org.apache.hadoop.yarn.util.resource.DominantResourceCalculator</value>
That config file might not even be available on all the gateways as its only
needed on RM.
> Spark on YARN does not work --executor-cores is specified
> ---------------------------------------------------------
>
> Key: SPARK-6050
> URL: https://issues.apache.org/jira/browse/SPARK-6050
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.3.0
> Environment: 2.5 based YARN cluster.
> Reporter: Mridul Muralidharan
> Priority: Blocker
>
> There are multiple issues here (which I will detail as comments), but to
> reproduce running the following ALWAYS hangs in our cluster with the 1.3 RC
> ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master
> yarn-cluster --executor-cores 8 --num-executors 15 --driver-memory 4g
> --executor-memory 2g --queue webmap lib/spark-examples*.jar
> 10
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
