hi, We're running spark 1.4.0 on ec2, with 6 machines, 4 cores each. We're trying to run an application on a number of total-executor-cores. but we want it to run on the minimal number of machines as possible (e.g. total-executor-cores=4, we'll want single machine. total-executor-cores=12, we'll want 3 machines)
I'm running spark shell, in the following command: /root/spark/bin/spark-shell --total-executor-cores X --executor-cores 4 or /root/spark/bin/spark-shell --total-executor-cores X and checked the cores on the spark UI, and found the following: Req total-executor-cores Actual cores with executor-cores param Actual cores without executor-cores=4 param 24 24 24 22 22 16 20 20 8 16 16 0 12 12 0 8 8 0 4 4 0 our questions: 1) Why we don't always get the number of cores we asked for when passing the "executor-cores 4" parameter? It seems that the number of cores we actually get is something like "max(24-(24-REQ_TOTAL_CORES)*4, 0)" 2) How can we get our original request? get the cores in minimal number of machines? When playing with the executor-cores, we have the problem described in (1), but the cores are on minimal number of cores 3) Playing with the parameter spark.deploy.spreadOut didn't seem to help with our request thanks, nizan -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/cores-and-resource-management-tp23628.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
