Hi Mark, if you know your cluster's number of workers and cores per worker you can set this up when you create a SparkContext and shouldn't need to tinker with the 'spark.executor.cores' setting. That setting is for running multiple executors per application per worker, which you are saying you don't want.
How to do what I'm describing? In standalone mode, you will be assigned cores in round robin order through the cluster's available workers (someone correct me if that has changed since 1.3). So if you have 4 workers and set `spark.cores.max` to `16` on your SparkContext then you will have 4 executors on each worker that are using 4 cores each. If you set `spark.cores.max` to `6` then two executors would have 2 cores and two executors would have 1 core. Hope that helps On Fri, Oct 23, 2015 at 3:05 PM, mkhaitman <mark.khait...@chango.com> wrote: > Regarding the 'spark.executor.cores' config option in a Standalone spark > environment, I'm curious about whether there's a way to enforce the > following logic: > > *- Max cores per executor = 4* > ** Max executors PER application PER worker = 1* > > In order to force better balance across all workers, I want to ensure that > a > single spark job can only ever use a specific upper limit on the number of > cores for each executor it holds, however, do not want a situation where it > can spawn 3 executors on a worker and only 1/2 on the others. Some spark > jobs end up using much more memory during aggregation tasks (joins / > groupBy's) which is more heavily impacted by the number of cores per > executor for that job. > > If this kind of setup/configuration doesn't already exist for Spark, and > others see the benefit of what I mean by this, where would be the best > location to insert this logic? > > Mark. > > > > -- > View this message in context: > http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-Executor-Cores-question-tp14763.html > Sent from the Apache Spark Developers List mailing list archive at > Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > > -- *Richard Marscher* Software Engineer Localytics Localytics.com <http://localytics.com/> | Our Blog <http://localytics.com/blog> | Twitter <http://twitter.com/localytics> | Facebook <http://facebook.com/localytics> | LinkedIn <http://www.linkedin.com/company/1148792?trk=tyah>
