[
https://issues.apache.org/jira/browse/MESOS-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14612267#comment-14612267
]
Marco Massenzio commented on MESOS-2985:
----------------------------------------
This is a bug due to the new workflow implementation.
We are hoping the INFRA guys can help us sort this out soon, if you do know
anyone and/or have any pull, help would be appreciated in raising the sense of
urgency :)
Tracked here:
INFRA-9846
INFRA-9903
> Wrong spark.executor.memory when using different EC2 master and worker
> machine types
> ------------------------------------------------------------------------------------
>
> Key: MESOS-2985
> URL: https://issues.apache.org/jira/browse/MESOS-2985
> Project: Mesos
> Issue Type: Bug
> Components: ec2
> Reporter: Stefano Parmesan
>
> _(this is a mirror of
> [SPARK-8726|https://issues.apache.org/jira/browse/SPARK-8726])_
> By default, {{spark.executor.memory}} is set to the [min(slave_ram_kb,
> master_ram_kb);|https://github.com/mesos/spark-ec2/blob/e642aa362338e01efed62948ec0f063d5fce3242/deploy_templates.py#L32]
> when using the same instance type for master and workers you will not
> notice, but when using different ones (which makes sense, as the master
> cannot be a spot instance, and using a big machine for the master would be a
> waste of resources) the default amount of memory given to each worker is
> capped to the amount of RAM available on the master (ex: if you create a
> cluster with an m1.small master (1.7GB RAM) and one m1.large worker (7.5GB
> RAM), spark.executor.memory will be set to 512MB).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
