On Wed, Aug 12, 2009 at 1:27 PM, Mayuran Yogarajah < [email protected]> wrote:
> Hello, > > Amandeep Khurana wrote: > >> On Wed, Aug 12, 2009 at 1:15 PM, Mayuran Yogarajah < >> [email protected]> wrote: >> >> >> >>> I had 3 jobs running and I saw something a bit odd. Two of the tasks >>> are reducing, one of them is using all the reducers so the other is >>> waiting, this is OK. However the 3rd job is still in the mapping phase >>> and even though the web interface shows map capacity at 96 I only >>> see about 7-12 mappers actually running. I'm wondering if theres >>> some setting I need to change, perhaps I've hit some system limit. Can >>> someone point me in the right direction please? >>> >>> >>> >> >> Are there any pending mappers remaining? Are you using any scheduler? >> >> >> > Yes there were pending mappers remaining, I'm not using any scheduler. > > >> >>> The other thing was that with the two jobs that are in the reducing >>> phase, >>> the reducer for one job wouldn't actually start until all the mappers of >>> the >>> _other_ job completed which seems kind of odd. Is this expected ? >>> >>> >>> >> >> Reducers dont really start the "reduce" phase till the mappers are >> completed. However, the process gets spawned off and the copying of the >> intermediate keys from the mapper starts off. >> >> >> >> > That was my understanding for the same job but this was across two > different jobs. There > were no reduce tasks running from job #2 until all of the map jobs of job > #1 completed. > > On a side note I just saw this in the task tracker log, I don't know if its > related: > INFO org.mortbay.http.SocketListener: LOW ON THREADS ((40-40+0)<1) on > [email protected]:50060 > WARN org.mortbay.http.SocketListener: OUT OF THREADS: > [email protected]:50060 > Ah.. That might be the issue.. I dont know the solution to this.. Wait for someone else to answer. The mappers not starting could be because of this as well. Whats your cluster configuration? How many cpu's, RAM etc...? > > > thanks, > M > >
