Felix, So did you change anything except the network configuration? What did you do to fix the "networking issues"?
--Bobby Evans On 7/14/11 2:46 PM, "felix gao" <gre1...@gmail.com> wrote: recently we had some network issues with our cluster. this job used to take on few minute to complete and how it is taking over half hour. when looking at the jobtracker's log i see it slowly getting all the splits information (the list is not exhaustive) 2011-07-14 14:42:51,434 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201107141056_0005_m_002488 has split on node:/default-rack/x.com <http://x.com> 2011-07-14 14:42:56,465 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201107141056_0005_m_002489 has split on node:/default-rack/x.com <http://x.com> 2011-07-14 14:43:01,446 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201107141056_0019_m_000218 has split on node:/default-rack/x.com <http://x.com> 2011-07-14 14:43:01,466 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201107141056_0010_m_001703 has split on node:/default-rack/x.com <http://x.com> 2011-07-14 14:43:01,490 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201107141056_0005_m_002489 has split on node:/default-rack/x.com <http://x.com> 2011-07-14 14:43:06,469 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201107141056_0010_m_001703 has split on node:/default-rack/x.com <http://x.com> 2011-07-14 14:43:06,473 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201107141056_0019_m_000218 has split on node:/default-rack/x.com <http://x.com> 2011-07-14 14:43:06,473 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201107141056_0019_m_000219 has split on node:/default-rack/x.com <http://x.com> 2011-07-14 14:43:06,473 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201107141056_0019_m_000219 has split on node:/default-rack/x.com <http://x.com> 2011-07-14 14:43:11,500 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201107141056_0019_m_000220 has split on node:/default-rack/x.com <http://x.com> 2011-07-14 14:43:11,542 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201107141056_0005_m_002491 has split on node:/default-rack/x.com <http://x.com> 2011-07-14 14:43:16,526 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201107141056_0019_m_000224 has split on node:/default-rack/x.com <http://x.com> 2011-07-14 14:43:16,526 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201107141056_0019_m_000225 has split on node:/default-rack/x.com <http://x.com> 2011-07-14 14:43:16,567 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201107141056_0005_m_002491 has split on node:/default-rack/x.com <http://x.com> 2011-07-14 14:45:26,791 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201107141056_0025_m_000001 has split on node:/default-rack/x.com <http://x.com> 2011-07-14 14:45:28,696 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201107141056_0005_m_002509 has split on node:/default-rack/x.com <http://x.com> 2011-07-14 14:45:31,770 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201107141056_0010_m_001722 has split on node:/default-rack/x.com <http://x.com> 2011-07-14 14:45:31,815 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201107141056_0025_m_000002 has split on node:/default-rack/x.com <http://x.com> 250 mappers tooks about 25 min to run, 10min spent on generating the tasks. The question is what could have caused this slow down? Thanks, Felix
