Thanks Koert and Alexander
I think the yarn configuration parameters in yarn-site,xml are important.
For those I have
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<description>Amount of max physical memory, in MB, that can be allocated
for YARN containers.</description>
<value>8192</value>
</property>
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<description>Ratio between virtual memory to physical memory when
setting memory limits for containers</description>
<value>2.1</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<description>Maximum memory for each container</description>
<value>8192</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<description>Minimum memory for each container</description>
<value>2048</value>
</property>
However, I noticed that you Alexander have the following settings
yarn.nodemanager.resource.memory-mb = 54272
yarn.scheduler.maximum-allocation-mb = 54272
With 8 Spark executor cores that gives you 6GB of memory per core. As a
matter of interest how much memory and how many cores do you have for each
node?
Thanks
Dr Mich Talebzadeh
LinkedIn *
https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 11 March 2016 at 23:01, Alexander Pivovarov <[email protected]> wrote:
> Forgot to mention. To avoid unnecessary container termination add the
> following setting to yarn
>
> yarn.nodemanager.vmem-check-enabled = false
>
>
