Thanks for the replies everyone! I did the 3 things that Mark mentioned, re-built & deployed the .tar.gz, then did `bin/grid stop all` and `bin/grid start all`. But when I re-submitted the 3 hello-samza jobs along with my 2 new jobs, yarn still won't run my 5th job. The yarn web ui now shows Memory Used = 4 GB and Memory Total = 8 GB, but my job still sits at State = ACCEPTED.
When I tail deploy/yarn/logs/yarn-vagrant-resourcemanager-precise64.log I see this repeated continuously: https://gist.githubusercontent.com/zcox/0f2b260d29e18d40d038/raw/0d805bcd7d8fec5332756efc9c990679480df117/gistfile1.txt I notice it says "available=<memory:4096, vCores:0>" - is my job not being run now because vCores=0? I also updated Vagrantfile to use: samza.vm.provider :virtualbox do |vb| vb.memory = 4096 vb.cpus = 8 end Thanks, Zach On Mon, Oct 6, 2014 at 12:40 PM, Lukas Steiblys <[email protected]> wrote: > I'll add that if you check the YARN node application master container log > and see that the job is constantly restarting, you might need to increase > the container memory limit to 1024MB at least. Also, a good parameter to > play with in YARN is yarn.nodemanager.vmem-pmem-ratio. > > Lukas > > -----Original Message----- From: Mark Mindenhall > Sent: Monday, October 6, 2014 8:44 AM > To: [email protected] > Subject: Re: Problems running new jobs in hello-samza > > > Hi Zach, > > I’m also a relative newbie, but I did run into this same issue. You are > correct, in that your 5th job isn’t starting due to not enough resources > available in the cluster, so you need to reduce the resources required. > > First, in yarn-site.xml I switched over to the FairScheduler< > http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/ > FairScheduler.html>: > > <property> > <name>yarn.resourcemanager.scheduler.class</name> > <value>org.apache.hadoop.yarn.server.resourcemanager. > scheduler.fair.FairScheduler</value> > </property> > > I also added these two properties (yarn-site.xml) to control the amount of > memory allocated to each job: > > <property> > <name>yarn.scheduler.minimum-allocation-mb</name> > <value>256</value> > <description>Minimum limit of memory to allocate to each container > request at the Resource Manager.</description> > </property> > <property> > <name>yarn.scheduler.maximum-allocation-mb</name> > <value>512</value> > <description>Maximum limit of memory to allocate to each container > request at the Resource Manager.</description> > </property> > > Then, in each of my Samza properties files describing my jobs, I added the > following two settings: > > yarn.container.memory.mb=512 > yarn.am.container.memory.mb=256 > > Hope that helps! > > Best, > Mark > > > On Oct 6, 2014, at 6:27 AM, Zach Cox <[email protected]<mailto:zcox > [email protected]>> wrote: > > Hi - I'm just getting started with Samza. I got the hello-samza example > working properly in the vagrant box. Then I wrote 2 new tasks, rebuilt > everything and submitted them to yarn using run-job.sh. These 2 new jobs > show up in the yarn web ui, however only one of them has State=RUNNING, the > other just sits forever at State=ACCEPTED. > > The Cluster Metrics section shows some interesting things: > - Apps Pending = 1 > - Apps Running = 4 > - Containers Running = 8 > - Memory Used = 8 GB > - Memory Total = 8 GB > - Memory Reserved = 0 B > > Again I'm really new to samza & yarn, but does this mean that the node on > this vagrant box has 8 GB memory available but all 8 GB is being used, so > it can't run the 5th samza job? > > Are there 8 containers running because each Samza job has an > ApplicationMaster and a SamzaContainer? Are each of those containers using > 1 GB memory, and that's why all the available memory is used up? Do these > containers really need 1 GB memory each? Can this be adjusted somehow? > > Just trying to better understand what's going on here, and see if there's a > simple way to get both of my new tasks running in hello-samza. > > Thanks, > Zach > >
