Hi Prateek,
Sorry, meant to include these versions with my email; I'm running
Samza 0.14 and Hadoop 2.6.1. I'm running three containers across 3
node managers, each with 16GB and 8 vcores. The other two containers
are requesting 1 vcore each; even with the AMs running, that should be
4 for them in total, leaving plenty of processing power available.
The error is in the application attempt diagnostics field: "The YARN
cluster is unable to run your job due to unsatisfiable resource
requirements. You asked for mem: 2048, and cpu: 2." I do not see this
error with the same memory request, but a cpu count request of 1.
Here are the configuration options pertaining to resource allocation:
<?xml version="1.0"?>
<configuration>
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler</value>
</property>
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>2.1</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>14336</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>256</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>14336</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-vcores</name>
<value>1</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-vcores</name>
<value>16</value>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>8</value>
</property>
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>processor-cluster</value>
</property>
</configuration>
Cheers,
Malcolm
On Mon, Apr 1, 2019 at 10:25 AM Prateek Maheshwari <prateek...@gmail.com> wrote:
>
> Hi Malcolm,
>
> Just setting that configuration should be sufficient. We haven't seen this
> issue before. What Samza/YARN versions are you using? Can you also include
> the logs from where you get the error and your yarn configuration?
>
> - Prateek
>
> On Mon, Apr 1, 2019 at 2:33 AM Malcolm McFarland <mmcfarl...@cavulus.com>
> wrote:
>
> > Hey Folks,
> >
> > I'm having some issues getting multiple cores for containers in yarn.
> > I seem to have my YARN settings correct, and the RM interface says
> > that I have 24vcores available. However, when I set the
> > cluster-manager.container.cpu.cores Samza setting to anything other
> > than 1, I get a message about how the container is requesting more
> > resources than it can allocate. With 1 core, everything is fine. Is
> > there another Samza option I need to set?
> >
> > Cheers,
> > Malcolm
> >
> >
> > --
> > Malcolm McFarland
> > Cavulus
> >
--
Malcolm McFarland
Cavulus
1-800-760-6915
mmcfarl...@cavulus.com
This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
unauthorized or improper disclosure, copying, distribution, or use of
the contents of this message is prohibited. The information contained
in this message is intended only for the personal and confidential use
of the recipient(s) named above. If you have received this message in
error, please notify the sender immediately and delete the original
message.
