Re: Mesos scheduler obeying limit of tasks / executor

[Iulian Dragoș]

Hi Charles,

I am not sure I totally understand your issues, but the spark.task.cpus
limit is imposed at a higher level, for all cluster managers. The code is
in TaskSchedulerImpl

.

There is a pending PR to implement spark.executor.cores (and launching
multiple executors on a single worker), but it wasn’t yet merged:
https://github.com/apache/spark/pull/4027

iulian
​

On Wed, Dec 9, 2015 at 7:23 PM, Charles Allen  wrote:

> I have a spark app in development which has relatively strict cpu/mem
> ratios that are required. As such, I cannot arbitrarily add CPUs to a
> limited memory size.
>
> The general spark cluster behaves as expected, where tasks are launched
> with a specified memory/cpu ratio, but the mesos scheduler seems to ignore
> this.
>
> Specifically, I cannot find where in the code the limit of number of tasks
> per executor of "spark.executor.cores" / "spark.task.cpus" is enforced in
> the MesosBackendScheduler.
>
> The Spark App in question has some JVM heap heavy activities inside a
> RDD.mapPartitionsWithIndex, so having more tasks per limited JVM memory
> resource is bad. The workaround planned handling of this is to limit the
> number of tasks per JVM, which does not seem possible in mesos mode, where
> it seems to just keep stacking on CPUs as tasks come in without adjusting
> any memory constraints, or looking for limits of tasks per executor.
>
> How can I limit the tasks per executor (or per memory pool) in the Mesos
> backend scheduler?
>
> Thanks,
> Charles Allen
>



-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: Mesos scheduler obeying limit of tasks / executor

[Charles Allen]

That answers it thanks!

On Fri, Dec 11, 2015 at 6:37 AM Iulian Dragoș 
wrote:

> Hi Charles,
>
> I am not sure I totally understand your issues, but the spark.task.cpus
> limit is imposed at a higher level, for all cluster managers. The code is
> in TaskSchedulerImpl
> 
> .
>
> There is a pending PR to implement spark.executor.cores (and launching
> multiple executors on a single worker), but it wasn’t yet merged:
> https://github.com/apache/spark/pull/4027
>
> iulian
> ​
>
> On Wed, Dec 9, 2015 at 7:23 PM, Charles Allen <
> charles.al...@metamarkets.com> wrote:
>
>> I have a spark app in development which has relatively strict cpu/mem
>> ratios that are required. As such, I cannot arbitrarily add CPUs to a
>> limited memory size.
>>
>> The general spark cluster behaves as expected, where tasks are launched
>> with a specified memory/cpu ratio, but the mesos scheduler seems to ignore
>> this.
>>
>> Specifically, I cannot find where in the code the limit of number of
>> tasks per executor of "spark.executor.cores" / "spark.task.cpus" is
>> enforced in the MesosBackendScheduler.
>>
>> The Spark App in question has some JVM heap heavy activities inside a
>> RDD.mapPartitionsWithIndex, so having more tasks per limited JVM memory
>> resource is bad. The workaround planned handling of this is to limit the
>> number of tasks per JVM, which does not seem possible in mesos mode, where
>> it seems to just keep stacking on CPUs as tasks come in without adjusting
>> any memory constraints, or looking for limits of tasks per executor.
>>
>> How can I limit the tasks per executor (or per memory pool) in the Mesos
>> backend scheduler?
>>
>> Thanks,
>> Charles Allen
>>
>
>
>
> --
>
> --
> Iulian Dragos
>
> --
> Reactive Apps on the JVM
> www.typesafe.com
>
>

Mesos scheduler obeying limit of tasks / executor

[Charles Allen]

I have a spark app in development which has relatively strict cpu/mem
ratios that are required. As such, I cannot arbitrarily add CPUs to a
limited memory size.

The general spark cluster behaves as expected, where tasks are launched
with a specified memory/cpu ratio, but the mesos scheduler seems to ignore
this.

Specifically, I cannot find where in the code the limit of number of tasks
per executor of "spark.executor.cores" / "spark.task.cpus" is enforced in
the MesosBackendScheduler.

The Spark App in question has some JVM heap heavy activities inside a
RDD.mapPartitionsWithIndex, so having more tasks per limited JVM memory
resource is bad. The workaround planned handling of this is to limit the
number of tasks per JVM, which does not seem possible in mesos mode, where
it seems to just keep stacking on CPUs as tasks come in without adjusting
any memory constraints, or looking for limits of tasks per executor.

How can I limit the tasks per executor (or per memory pool) in the Mesos
backend scheduler?

Thanks,
Charles Allen