Hello,
Amandeep Khurana wrote:
On Wed, Aug 12, 2009 at 1:15 PM, Mayuran Yogarajah <
[email protected]> wrote:
I had 3 jobs running and I saw something a bit odd. Two of the tasks
are reducing, one of them is using all the reducers so the other is
waiting, this is OK. However the 3rd job is still in the mapping phase
and even though the web interface shows map capacity at 96 I only
see about 7-12 mappers actually running. I'm wondering if theres
some setting I need to change, perhaps I've hit some system limit. Can
someone point me in the right direction please?
Are there any pending mappers remaining? Are you using any scheduler?
Yes there were pending mappers remaining, I'm not using any scheduler.
The other thing was that with the two jobs that are in the reducing phase,
the reducer for one job wouldn't actually start until all the mappers of
the
_other_ job completed which seems kind of odd. Is this expected ?
Reducers dont really start the "reduce" phase till the mappers are
completed. However, the process gets spawned off and the copying of the
intermediate keys from the mapper starts off.
That was my understanding for the same job but this was across two
different jobs. There
were no reduce tasks running from job #2 until all of the map jobs of
job #1 completed.
On a side note I just saw this in the task tracker log, I don't know if
its related:
INFO org.mortbay.http.SocketListener: LOW ON THREADS ((40-40+0)<1) on
[email protected]:50060
WARN org.mortbay.http.SocketListener: OUT OF THREADS:
[email protected]:50060
thanks,
M
