That's correct. There were some changes last year as to. How these worker containers are allocated that were supposed to fix another similar issue with the master task. Perhaps this introduced another bug. I'm not sure there's enough info to be sure can you give more details? Anyone else seeing this issue? And to confirm this was only when you run on a single machine? On Jan 17, 2015 12:09 PM, "Claudio Martella" <[email protected]> wrote:
> as far as I understand, with YARN there are two additional task to the > workers: one for the master and one for the application manager. I'm not > sure, but maybe Eli can confirm. > > On Fri, Jan 16, 2015 at 2:16 PM, Dongjin Lee <[email protected]> > wrote: > > > * I apologize in advance for my bad English. > > > > Hello. I have a question about determining optimal worker count. I have > > been trying SimpleShortestPathsComputation example with Hadoop 2.5.1. And > > noticed that the maximum value for -w parameter differs per > configuration. > > > > Let me explain: I have been running Giraph jobs on my local Ubuntu > machine, > > with Intel i7 cpu (4core) and 16gb ram. I can run 8 threads > simultaneously > > and has 9gb of free memory. > > > > First, I configured Hadoop can run 4 mappers with 2gb of memory for each. > > When I ran SimpleShortestPathsComputation, w=2 worked well but w=3 did > not > > work - It froze with 'map 75%'. That is, I could not use 2 mappers for > > worker. > > > > After that, I re-configured my Hadoop distribution to run 8 mappers with > > 1gb of memory for each. When I re-ran SimpleShortestPathsComputation, w=5 > > worked well but with w=6 it froze with 'map 86%'. In this case, I could > not > > use 3 mappers for worker. > > > > I already know that there is an additional mapper, which works as > > BSPMaster. However, It is still mysterious for me: Is there any hidden > > process I don't know? Then, how many are there? 1 or 2? > > > > Any guidance will be welcomed. > > > > Thanks in advance, > > Dongjin > > > > PS - When I was working with Hadoop 1.x.x, I calculated the maximum > worker > > count as {available mappers} - 1 and it worked perfectly. I guess there > are > > some differences between 1.x.x and 2.x.x that I don't know. > > > > -- > > *Dongjin Lee* > > > > > > *Oracle Server Technologies Group.So interested in massive-scale machine > > learning.facebook: www.facebook.com/dongjin.lee.kr > > <http://www.facebook.com/dongjin.lee.kr>linkedin: > > kr.linkedin.com/in/dongjinleekr > > <http://kr.linkedin.com/in/dongjinleekr>github: > > <http://goog_969573159>github.com/dongjinleekr > > <http://github.com/dongjinleekr>twitter: www.twitter.com/dongjinleekr > > <http://www.twitter.com/dongjinleekr>* > > > > > > -- > Claudio Martella >
