Yes, you need to increase the core count in yarn
(yarn.nodemanager.resource.cpu-vcores). In your case it seems like you need
at least 10 cores for 5 jobs.
Lukas
-----Original Message-----
From: Zach Cox
Sent: Monday, October 6, 2014 11:27 AM
To: [email protected]
Subject: Re: Problems running new jobs in hello-samza
Thanks for the replies everyone! I did the 3 things that Mark mentioned,
re-built & deployed the .tar.gz, then did `bin/grid stop all` and `bin/grid
start all`. But when I re-submitted the 3 hello-samza jobs along with my 2
new jobs, yarn still won't run my 5th job. The yarn web ui now shows Memory
Used = 4 GB and Memory Total = 8 GB, but my job still sits at State =
ACCEPTED.
When I tail deploy/yarn/logs/yarn-vagrant-resourcemanager-precise64.log I
see this repeated continuously:
https://gist.githubusercontent.com/zcox/0f2b260d29e18d40d038/raw/0d805bcd7d8fec5332756efc9c990679480df117/gistfile1.txt
I notice it says "available=<memory:4096, vCores:0>" - is my job not being
run now because vCores=0?
I also updated Vagrantfile to use:
samza.vm.provider :virtualbox do |vb| vb.memory = 4096 vb.cpus = 8 end
Thanks,
Zach
On Mon, Oct 6, 2014 at 12:40 PM, Lukas Steiblys <[email protected]>
wrote:
I'll add that if you check the YARN node application master container log
and see that the job is constantly restarting, you might need to increase
the container memory limit to 1024MB at least. Also, a good parameter to
play with in YARN is yarn.nodemanager.vmem-pmem-ratio.
Lukas
-----Original Message----- From: Mark Mindenhall
Sent: Monday, October 6, 2014 8:44 AM
To: [email protected]
Subject: Re: Problems running new jobs in hello-samza
Hi Zach,
I’m also a relative newbie, but I did run into this same issue. You are
correct, in that your 5th job isn’t starting due to not enough resources
available in the cluster, so you need to reduce the resources required.
First, in yarn-site.xml I switched over to the FairScheduler<
http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/
FairScheduler.html>:
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.
scheduler.fair.FairScheduler</value>
</property>
I also added these two properties (yarn-site.xml) to control the amount of
memory allocated to each job:
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>256</value>
<description>Minimum limit of memory to allocate to each container
request at the Resource Manager.</description>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>512</value>
<description>Maximum limit of memory to allocate to each container
request at the Resource Manager.</description>
</property>
Then, in each of my Samza properties files describing my jobs, I added the
following two settings:
yarn.container.memory.mb=512
yarn.am.container.memory.mb=256
Hope that helps!
Best,
Mark
On Oct 6, 2014, at 6:27 AM, Zach Cox <[email protected]<mailto:zcox
[email protected]>> wrote:
Hi - I'm just getting started with Samza. I got the hello-samza example
working properly in the vagrant box. Then I wrote 2 new tasks, rebuilt
everything and submitted them to yarn using run-job.sh. These 2 new jobs
show up in the yarn web ui, however only one of them has State=RUNNING,
the
other just sits forever at State=ACCEPTED.
The Cluster Metrics section shows some interesting things:
- Apps Pending = 1
- Apps Running = 4
- Containers Running = 8
- Memory Used = 8 GB
- Memory Total = 8 GB
- Memory Reserved = 0 B
Again I'm really new to samza & yarn, but does this mean that the node on
this vagrant box has 8 GB memory available but all 8 GB is being used, so
it can't run the 5th samza job?
Are there 8 containers running because each Samza job has an
ApplicationMaster and a SamzaContainer? Are each of those containers using
1 GB memory, and that's why all the available memory is used up? Do these
containers really need 1 GB memory each? Can this be adjusted somehow?
Just trying to better understand what's going on here, and see if there's
a
simple way to get both of my new tasks running in hello-samza.
Thanks,
Zach
