I have not found a way to control the cores yet. This effectively limits the cluster to a single application at a time. A subsequent application shows in the 'WAITING' State on the dashboard.
On Wed, Oct 1, 2014 at 2:49 PM, Akshat Aranya <aara...@gmail.com> wrote: > > > On Wed, Oct 1, 2014 at 11:33 AM, Akshat Aranya <aara...@gmail.com> wrote: > >> >> >> On Wed, Oct 1, 2014 at 11:00 AM, Boromir Widas <vcsub...@gmail.com> >> wrote: >> >>> 1. worker memory caps executor. >>> 2. With default config, every job gets one executor per worker. This >>> executor runs with all cores available to the worker. >>> >>> By the job do you mean one SparkContext or one stage execution within a >> program? Does that also mean that two concurrent jobs will get one >> executor each at the same time? >> > > Experimenting with this some more, I figured out that an executor takes > away "spark.executor.memory" amount of memory from the configured worker > memory. It also takes up all the cores, so even if there is still some > memory left, there are no cores left for starting another executor. Is my > assessment correct? Is there no way to configure the number of cores that > an executor can use? > > >> >>> >>> On Wed, Oct 1, 2014 at 11:04 AM, Akshat Aranya <aara...@gmail.com> >>> wrote: >>> >>>> Hi, >>>> >>>> What's the relationship between Spark worker and executor memory >>>> settings in standalone mode? Do they work independently or does the worker >>>> cap executor memory? >>>> >>>> Also, is the number of concurrent executors per worker capped by the >>>> number of CPU cores configured for the worker? >>>> >>> >>> >> >
