[jira] [Commented] (SPARK-19090) Dynamic Resource Allocation not respecting spark.executor.cores

[Carlos Vicenti (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16157932#comment-16157932
 ] 

Carlos Vicenti commented on SPARK-19090:


I have found the same issue while using Hive On Spark (on Yarn) and 
spark.dynamicAllocation.enabled set to true
{noformat}
SET spark.executor.cores=4;
SET spark.executor.memory=21G;
SET spark.yarn.executor.memoryOverhead=3813;
{noformat}

>From the application logs:
{noformat}
17/09/08 00:30:34 INFO yarn.YarnAllocator: Will request 1 executor containers, 
each with 6 cores and 25317 MB memory including 3813 MB overhead
{noformat}

As mentioned above. This does not happen if I set 
spark.dynamicAllocation.enabled to false

> Dynamic Resource Allocation not respecting spark.executor.cores
> ---
>
> Key: SPARK-19090
> URL: https://issues.apache.org/jira/browse/SPARK-19090
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 1.5.2, 1.6.1, 2.0.1
>Reporter: nirav patel
>
> When enabling dynamic scheduling with yarn I see that all executors are using 
> only 1 core even if I specify "spark.executor.cores" to 6. If dynamic 
> scheduling is disabled then each executors will have 6 cores. i.e. it 
> respects  "spark.executor.cores". I have tested this against spark 1.5 . I 
> think it will be the same behavior with 2.x as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-19090) Dynamic Resource Allocation not respecting spark.executor.cores

[nirav patel (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818716#comment-15818716
 ] 

nirav patel commented on SPARK-19090:
-

[~q79969786] As I mentioned in previous comment it does work for me when I set 
parameter on command line but it doesn't work when I set it via SparkConf in my 
application class.

> Dynamic Resource Allocation not respecting spark.executor.cores
> ---
>
> Key: SPARK-19090
> URL: https://issues.apache.org/jira/browse/SPARK-19090
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 1.5.2, 1.6.1, 2.0.1
>Reporter: nirav patel
>
> When enabling dynamic scheduling with yarn I see that all executors are using 
> only 1 core even if I specify "spark.executor.cores" to 6. If dynamic 
> scheduling is disabled then each executors will have 6 cores. i.e. it 
> respects  "spark.executor.cores". I have tested this against spark 1.5 . I 
> think it will be the same behavior with 2.x as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-19090) Dynamic Resource Allocation not respecting spark.executor.cores

[Yuming Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816920#comment-15816920
 ] 

Yuming Wang commented on SPARK-19090:
-

Try this, it works for me:
{code}
sbin/start-thriftserver.sh --executor-memory 12g --driver-memory 8g 
--executor-cores 5 --num-executors 130 --hiveconf 
hive.server2.thrift.port=20402  --conf 
spark.scheduler.listenerbus.eventqueue.size=15 --conf 
spark.ui.retainedTasks=20 --conf spark.sql.codegen.wholeStage=true --conf 
spark.dynamicAllocation.enabled=true --conf spark.shuffle.service.enabled=true 
--conf spark.dynamicAllocation.maxExecutors=130 --conf 
spark.dynamicAllocation.executorIdleTimeout=200 --hiveconf 
hive.server2.thrift.bind.host=192.168.28.200 --conf 
spark.yarn.executor.memoryOverhead=4096 --conf 
"spark.executor.extraJavaOptions=-XX:+UseParallelGC -XX:+UseParallelOldGC 
-XX:+PrintFlagsFinal -XX:+PrintReferenceGC -verbose:gc -XX:+PrintGCDetails 
-XX:+PrintGCTimeStamps -XX:+PrintAdaptiveSizePolicy 
-XX:+UnlockDiagnosticVMOptions"
{code}

> Dynamic Resource Allocation not respecting spark.executor.cores
> ---
>
> Key: SPARK-19090
> URL: https://issues.apache.org/jira/browse/SPARK-19090
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 1.5.2, 1.6.1, 2.0.1
>Reporter: nirav patel
>
> When enabling dynamic scheduling with yarn I see that all executors are using 
> only 1 core even if I specify "spark.executor.cores" to 6. If dynamic 
> scheduling is disabled then each executors will have 6 cores. i.e. it 
> respects  "spark.executor.cores". I have tested this against spark 1.5 . I 
> think it will be the same behavior with 2.x as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-19090) Dynamic Resource Allocation not respecting spark.executor.cores

[nirav patel (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815580#comment-15815580
 ] 

nirav patel commented on SPARK-19090:
-

I tried passing spark parameters via oozie directly using   --conf 
spark.executor.cores=5 which passes those parameters to  : 
org.apache.spark.deploy.SparkSubmit and then SparkSubmit subsequently calls my 
application class. In this scenario I can see that dynamic executor scheduling 
picking up this value and using it to request 5 vcores per executor. SO I think 
that's my workaround. 
Real issue seems that spark dynamic scheduling module is ignoring sparkConf 
(spark.executor.cores) parameter set by user application class. It's 
recognizing all other parameters! Because I am not passing any other parameter 
directly to spark submit those are all set via my application code as shown in 
my code snippet. It's only ignoring spark.executor.cores set at application 
level which is weird. If I have read document correctly one can always override 
spark command line parameter via application level SparkConf object. It 
definitely works when dynamic scheduling is turned off . It only doesn't work 
when its on. 

> Dynamic Resource Allocation not respecting spark.executor.cores
> ---
>
> Key: SPARK-19090
> URL: https://issues.apache.org/jira/browse/SPARK-19090
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 1.5.2, 1.6.1, 2.0.1
>Reporter: nirav patel
>
> When enabling dynamic scheduling with yarn I see that all executors are using 
> only 1 core even if I specify "spark.executor.cores" to 6. If dynamic 
> scheduling is disabled then each executors will have 6 cores. i.e. it 
> respects  "spark.executor.cores". I have tested this against spark 1.5 . I 
> think it will be the same behavior with 2.x as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-19090) Dynamic Resource Allocation not respecting spark.executor.cores

[nirav patel (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815510#comment-15815510
 ] 

nirav patel commented on SPARK-19090:
-

Yes, I am setting it from a Main application class. It works with all our jobs 
in yarn-client mode. We don't use yarn-cluster.

> Dynamic Resource Allocation not respecting spark.executor.cores
> ---
>
> Key: SPARK-19090
> URL: https://issues.apache.org/jira/browse/SPARK-19090
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 1.5.2, 1.6.1, 2.0.1
>Reporter: nirav patel
>
> When enabling dynamic scheduling with yarn I see that all executors are using 
> only 1 core even if I specify "spark.executor.cores" to 6. If dynamic 
> scheduling is disabled then each executors will have 6 cores. i.e. it 
> respects  "spark.executor.cores". I have tested this against spark 1.5 . I 
> think it will be the same behavior with 2.x as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-19090) Dynamic Resource Allocation not respecting spark.executor.cores

[Saisai Shao (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15814001#comment-15814001
 ] 

Saisai Shao commented on SPARK-19090:
-

Are you using SparkConf API to set configuration in application run-time? From 
the code I could see you did that . This won't be worked, at least for yarn 
cluster mode.

> Dynamic Resource Allocation not respecting spark.executor.cores
> ---
>
> Key: SPARK-19090
> URL: https://issues.apache.org/jira/browse/SPARK-19090
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 1.5.2, 1.6.1, 2.0.1
>Reporter: nirav patel
>
> When enabling dynamic scheduling with yarn I see that all executors are using 
> only 1 core even if I specify "spark.executor.cores" to 6. If dynamic 
> scheduling is disabled then each executors will have 6 cores. i.e. it 
> respects  "spark.executor.cores". I have tested this against spark 1.5 . I 
> think it will be the same behavior with 2.x as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-19090) Dynamic Resource Allocation not respecting spark.executor.cores

[Saisai Shao (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813973#comment-15813973
 ] 

Saisai Shao commented on SPARK-19090:
-

Spark shell is a real spark *application*. The underlying SparkSubmit logics 
are the same...

> Dynamic Resource Allocation not respecting spark.executor.cores
> ---
>
> Key: SPARK-19090
> URL: https://issues.apache.org/jira/browse/SPARK-19090
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 1.5.2, 1.6.1, 2.0.1
>Reporter: nirav patel
>
> When enabling dynamic scheduling with yarn I see that all executors are using 
> only 1 core even if I specify "spark.executor.cores" to 6. If dynamic 
> scheduling is disabled then each executors will have 6 cores. i.e. it 
> respects  "spark.executor.cores". I have tested this against spark 1.5 . I 
> think it will be the same behavior with 2.x as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-19090) Dynamic Resource Allocation not respecting spark.executor.cores

[nirav patel (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813969#comment-15813969
 ] 

nirav patel commented on SPARK-19090:
-

Also you are just invoking spark-shell here and not submitting any real spark 
application. Those two may have different code path that uses different logic 
to set spark parameters. 

> Dynamic Resource Allocation not respecting spark.executor.cores
> ---
>
> Key: SPARK-19090
> URL: https://issues.apache.org/jira/browse/SPARK-19090
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 1.5.2, 1.6.1, 2.0.1
>Reporter: nirav patel
>
> When enabling dynamic scheduling with yarn I see that all executors are using 
> only 1 core even if I specify "spark.executor.cores" to 6. If dynamic 
> scheduling is disabled then each executors will have 6 cores. i.e. it 
> respects  "spark.executor.cores". I have tested this against spark 1.5 . I 
> think it will be the same behavior with 2.x as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-19090) Dynamic Resource Allocation not respecting spark.executor.cores

[nirav patel (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813962#comment-15813962
 ] 

nirav patel commented on SPARK-19090:
-

Oh right, I have that set exclusively. I corrected my comment. I verified that 
dynamic allocation was enabled by checking following in driver logs:
2017-01-04 12:04:11,362 INFO [spark-dynamic-executor-allocation] 
org.apache.spark.ExecutorAllocationManager: Requesting 4 new executors because 
tasks are backlogged (new desired total will be 6)

If it was not enabled then it should have actually create 6 executors with 5 
cores. 

here's the snippet of code I have:

  if(sparkConfig.dynamicAllocation){

sparkConf.set("spark.dynamicAllocation.enabled", "true")

sparkConf.set("spark.dynamicAllocation.executorIdleTimeout", "600s")

sparkConf.set("spark.dynamicAllocation.initialExecutors", 
sparkConfig.executorInstances)

sparkConf.set("spark.dynamicAllocation.minExecutors", 
String.valueOf((Integer.valueOf(sparkConfig.executorInstances) - 3)))

sparkConf.set("spark.dynamicAllocation.sustainedSchedulerBacklogTimeout", 
"300s")

sparkConf.set("spark.dynamicAllocation.schedulerBacklogTimeout", "120")

} else {
  sparkConf.set("spark.executor.instances", 
sparkConfig.executorInstances)
}

  sparkConf.set("spark.executor.cores", 
sparkConfig.executorCores)

> Dynamic Resource Allocation not respecting spark.executor.cores
> ---
>
> Key: SPARK-19090
> URL: https://issues.apache.org/jira/browse/SPARK-19090
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 1.5.2, 1.6.1, 2.0.1
>Reporter: nirav patel
>
> When enabling dynamic scheduling with yarn I see that all executors are using 
> only 1 core even if I specify "spark.executor.cores" to 6. If dynamic 
> scheduling is disabled then each executors will have 6 cores. i.e. it 
> respects  "spark.executor.cores". I have tested this against spark 1.5 . I 
> think it will be the same behavior with 2.x as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-19090) Dynamic Resource Allocation not respecting spark.executor.cores

[Saisai Shao (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813941#comment-15813941
 ] 

Saisai Shao commented on SPARK-19090:
-

{code}
./bin/spark-shell --master yarn-client --conf spark.executor.cores=2
{code}

Please be aware that executor number (--num-executors/spark.executor.instances) 
and dynamic allocation cannot be coexisted, otherwise dynamic allocation will 
be turned off implicitly. So in your case you set executor numbers also, which 
means dynamic allocation is not on actually.

> Dynamic Resource Allocation not respecting spark.executor.cores
> ---
>
> Key: SPARK-19090
> URL: https://issues.apache.org/jira/browse/SPARK-19090
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 1.5.2, 1.6.1, 2.0.1
>Reporter: nirav patel
>
> When enabling dynamic scheduling with yarn I see that all executors are using 
> only 1 core even if I specify "spark.executor.cores" to 6. If dynamic 
> scheduling is disabled then each executors will have 6 cores. i.e. it 
> respects  "spark.executor.cores". I have tested this against spark 1.5 . I 
> think it will be the same behavior with 2.x as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-19090) Dynamic Resource Allocation not respecting spark.executor.cores

[nirav patel (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813918#comment-15813918
 ] 

nirav patel commented on SPARK-19090:
-

I am using oozie spark-action to submit job. I set all spark parameters via 
SparkConf. Can you tell me what parameter you used with dynamic allocation to 
allow number of cores = 2 ?

> Dynamic Resource Allocation not respecting spark.executor.cores
> ---
>
> Key: SPARK-19090
> URL: https://issues.apache.org/jira/browse/SPARK-19090
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 1.5.2, 1.6.1, 2.0.1
>Reporter: nirav patel
>
> When enabling dynamic scheduling with yarn I see that all executors are using 
> only 1 core even if I specify "spark.executor.cores" to 6. If dynamic 
> scheduling is disabled then each executors will have 6 cores. i.e. it 
> respects  "spark.executor.cores". I have tested this against spark 1.5 . I 
> think it will be the same behavior with 2.x as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-19090) Dynamic Resource Allocation not respecting spark.executor.cores

[Saisai Shao (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813801#comment-15813801
 ] 

Saisai Shao commented on SPARK-19090:
-

I also tested with Spark 1.5.0, I don't see an issue here, the core number is 
still expected as I set:

{noformat}
17/01/10 12:00:31 INFO yarn.YarnRMClient: Registering the ApplicationMaster
17/01/10 12:00:31 INFO yarn.YarnAllocator: Will request 1 executor containers, 
each with 2 cores and 1408 MB memory including 384 MB overhead
17/01/10 12:00:31 INFO yarn.YarnAllocator: Container request (host: Any, 
capability: )
17/01/10 12:00:31 INFO yarn.ApplicationMaster: Started progress reporter thread 
with (heartbeat : 3000, initial allocation : 200) intervals
{noformat}

Can you please tell how do you  run the application?




> Dynamic Resource Allocation not respecting spark.executor.cores
> ---
>
> Key: SPARK-19090
> URL: https://issues.apache.org/jira/browse/SPARK-19090
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 1.5.2, 1.6.1, 2.0.1
>Reporter: nirav patel
>
> When enabling dynamic scheduling with yarn I see that all executors are using 
> only 1 core even if I specify "spark.executor.cores" to 6. If dynamic 
> scheduling is disabled then each executors will have 6 cores. i.e. it 
> respects  "spark.executor.cores". I have tested this against spark 1.5 . I 
> think it will be the same behavior with 2.x as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-19090) Dynamic Resource Allocation not respecting spark.executor.cores

[Saisai Shao (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813782#comment-15813782
 ] 

Saisai Shao commented on SPARK-19090:
-

I tested with Spark 2.0 and latest master (2.2.0-SNAPSHOT), the behavior is 
correct, if there's an issue in it, it should be 1.x issue. I'm not sure are we 
still maintaining 1.5 branch to fix the old bug if existed.

> Dynamic Resource Allocation not respecting spark.executor.cores
> ---
>
> Key: SPARK-19090
> URL: https://issues.apache.org/jira/browse/SPARK-19090
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 1.5.2, 1.6.1, 2.0.1
>Reporter: nirav patel
>
> When enabling dynamic scheduling with yarn I see that all executors are using 
> only 1 core even if I specify "spark.executor.cores" to 6. If dynamic 
> scheduling is disabled then each executors will have 6 cores. i.e. it 
> respects  "spark.executor.cores". I have tested this against spark 1.5 . I 
> think it will be the same behavior with 2.x as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-19090) Dynamic Resource Allocation not respecting spark.executor.cores

[nirav patel (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813723#comment-15813723
 ] 

nirav patel commented on SPARK-19090:
-

1.5.2 

> Dynamic Resource Allocation not respecting spark.executor.cores
> ---
>
> Key: SPARK-19090
> URL: https://issues.apache.org/jira/browse/SPARK-19090
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 1.5.2, 1.6.1, 2.0.1
>Reporter: nirav patel
>
> When enabling dynamic scheduling with yarn I see that all executors are using 
> only 1 core even if I specify "spark.executor.cores" to 6. If dynamic 
> scheduling is disabled then each executors will have 6 cores. i.e. it 
> respects  "spark.executor.cores". I have tested this against spark 1.5 . I 
> think it will be the same behavior with 2.x as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-19090) Dynamic Resource Allocation not respecting spark.executor.cores

[Saisai Shao (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813704#comment-15813704
 ] 

Saisai Shao commented on SPARK-19090:
-

Thanks for your elaboration, would you please tell which version of Spark did 
you run the test?

> Dynamic Resource Allocation not respecting spark.executor.cores
> ---
>
> Key: SPARK-19090
> URL: https://issues.apache.org/jira/browse/SPARK-19090
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 1.5.2, 1.6.1, 2.0.1
>Reporter: nirav patel
>
> When enabling dynamic scheduling with yarn I see that all executors are using 
> only 1 core even if I specify "spark.executor.cores" to 6. If dynamic 
> scheduling is disabled then each executors will have 6 cores. i.e. it 
> respects  "spark.executor.cores". I have tested this against spark 1.5 . I 
> think it will be the same behavior with 2.x as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-19090) Dynamic Resource Allocation not respecting spark.executor.cores

[nirav patel (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813689#comment-15813689
 ] 

nirav patel commented on SPARK-19090:
-

[~jerryshao] 

"spark.executor.cores" is to tell spark AM to request no of vcores from Yarn 
per container. I think spark AM makes correct decision when dynamic allocation 
is off but when its on it ignores spark.executor.cores value. I think DRF has 
nothing to do with this in my opinion. Following are AM logs from two different 
runs. 


Run 1:
spark.dynamicAllocation.enabled = true
spark.executor.instances = 6
spark.executor.cores = 5 

Dynamic allocation = true

17/01/09 19:05:49 INFO yarn.YarnRMClient: Registering the ApplicationMaster
17/01/09 19:05:49 INFO yarn.YarnAllocator: Will request 6 executor containers, 
each with 1 cores and 11000 MB memory including 1000 MB overhead
17/01/09 19:05:49 INFO yarn.YarnAllocator: Container request (host: Any, 
capability: )
17/01/09 19:05:49 INFO yarn.YarnAllocator: Container request (host: Any, 
capability: )
17/01/09 19:05:49 INFO yarn.YarnAllocator: Container request (host: Any, 
capability: )
17/01/09 19:05:49 INFO yarn.YarnAllocator: Container request (host: Any, 
capability: )
17/01/09 19:05:49 INFO yarn.YarnAllocator: Container request (host: Any, 
capability: )
17/01/09 19:05:49 INFO yarn.YarnAllocator: Container request (host: Any, 
capability: )


Run 2:

spark.dynamicAllocation.enabled = false
spark.executor.instances = 6
spark.executor.cores = 5 

17/01/09 19:01:39 INFO yarn.YarnAllocator: Will request 6 executor containers, 
each with 5 cores and 11000 MB memory including 1000 MB overhead
17/01/09 19:01:39 INFO yarn.YarnAllocator: Container request (host: Any, 
capability: )
17/01/09 19:01:39 INFO yarn.YarnAllocator: Container request (host: Any, 
capability: )
17/01/09 19:01:39 INFO yarn.YarnAllocator: Container request (host: Any, 
capability: )
17/01/09 19:01:39 INFO yarn.YarnAllocator: Container request (host: Any, 
capability: )
17/01/09 19:01:39 INFO yarn.YarnAllocator: Container request (host: Any, 
capability: )
17/01/09 19:01:39 INFO yarn.YarnAllocator: Container request (host: Any, 
capability: )

I can verify same fact via Spark UI when job is running that with dynamic 
allocation there is only 1 task running per executor.

> Dynamic Resource Allocation not respecting spark.executor.cores
> ---
>
> Key: SPARK-19090
> URL: https://issues.apache.org/jira/browse/SPARK-19090
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 1.5.2, 1.6.1, 2.0.1
>Reporter: nirav patel
>
> When enabling dynamic scheduling with yarn I see that all executors are using 
> only 1 core even if I specify "spark.executor.cores" to 6. If dynamic 
> scheduling is disabled then each executors will have 6 cores. i.e. it 
> respects  "spark.executor.cores". I have tested this against spark 1.5 . I 
> think it will be the same behavior with 2.x as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-19090) Dynamic Resource Allocation not respecting spark.executor.cores

[Saisai Shao (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15803754#comment-15803754
 ] 

Saisai Shao commented on SPARK-19090:
-

"spark.executor.cores" has nothing to do with dynamic allocation, dynamic 
allocation schedules executors not cores. I guess the problem you saw, core 
number is always 1, is due to Dominant Resource Fairness in yarn, please enable 
this if you want cpu scheduling. Otherwise the core number will always be 1 
seen from yarn side.

> Dynamic Resource Allocation not respecting spark.executor.cores
> ---
>
> Key: SPARK-19090
> URL: https://issues.apache.org/jira/browse/SPARK-19090
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 1.5.2, 1.6.1, 2.0.1
>Reporter: nirav patel
>
> When enabling dynamic scheduling with yarn I see that all executors are using 
> only 1 core even if I specify "spark.executor.cores" to 6. If dynamic 
> scheduling is disabled then each executors will have 6 cores. i.e. it 
> respects  "spark.executor.cores". I have tested this against spark 1.5 . I 
> think it will be the same behavior with 2.x as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org