I just run simple map reduce task. (identity mapper, identity reducer) Job progress reach map 100% reduce 89~92%... Job hanging... what's wrong.
version : hadoop 0.8.0 os : FreeBSD 6.2 64bit data size : 886M data type : org.apache.nutch.protocol.Content box size : 9 (1 jobtracker box, 7 tasktracker box, all datanode, namenode running another box) mapred.map.tasks : 140 mapred.reduce.tasks : 14 thread dump : 2007-02-15 10:14:11,904 INFO mapred.TaskTracker - Process Thread Dump: lost task 20 active threads Thread 320 (IPC Client connection to xc1/10.8.50.31:9001): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: org.apache.hadoop.ipc.Client$Call.callComplete(Client.java:89) org.apache.hadoop.ipc.Client$Connection.run(Client.java:265) Thread 258 (Thread-223): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: java.io.FileInputStream.readBytes(Native Method) java.io.FileInputStream.read(FileInputStream.java:194) sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411) sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453) sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183) java.io.InputStreamReader.read(InputStreamReader.java:167) java.io.BufferedReader.fill(BufferedReader.java:136) java.io.BufferedReader.readLine(BufferedReader.java:299) java.io.BufferedReader.readLine(BufferedReader.java:362) org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340) org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36) org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304) Thread 257 (process reaper): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: java.lang.UNIXProcess.waitForProcessExit(Native Method) java.lang.UNIXProcess.access$900(UNIXProcess.java:20) java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132) Thread 255 (Thread-221): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: java.io.FileInputStream.readBytes(Native Method) java.io.FileInputStream.read(FileInputStream.java:194) sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411) sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453) sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183) java.io.InputStreamReader.read(InputStreamReader.java:167) java.io.BufferedReader.fill(BufferedReader.java:136) java.io.BufferedReader.readLine(BufferedReader.java:299) java.io.BufferedReader.readLine(BufferedReader.java:362) org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340) org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36) org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304) Thread 254 (process reaper): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: java.lang.UNIXProcess.waitForProcessExit(Native Method) java.lang.UNIXProcess.access$900(UNIXProcess.java:20) java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132) Thread 239 (Thread-206): State: RUNNABLE Blocked count: 36 Waited count: 147 Stack: java.io.FileInputStream.readBytes(Native Method) java.io.FileInputStream.read(FileInputStream.java:194) java.io.BufferedInputStream.read1(BufferedInputStream.java:254) java.io.BufferedInputStream.read(BufferedInputStream.java:313) sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411) sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453) sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183) java.io.InputStreamReader.read(InputStreamReader.java:167) java.io.BufferedReader.fill(BufferedReader.java:136) java.io.BufferedReader.readLine(BufferedReader.java:299) java.io.BufferedReader.readLine(BufferedReader.java:362) org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340) org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308) org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198) Thread 230 (Thread-198): State: RUNNABLE Blocked count: 30 Waited count: 141 Stack: java.io.FileInputStream.readBytes(Native Method) java.io.FileInputStream.read(FileInputStream.java:194) java.io.BufferedInputStream.read1(BufferedInputStream.java:254) java.io.BufferedInputStream.read(BufferedInputStream.java:313) sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411) sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453) sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183) java.io.InputStreamReader.read(InputStreamReader.java:167) java.io.BufferedReader.fill(BufferedReader.java:136) java.io.BufferedReader.readLine(BufferedReader.java:299) java.io.BufferedReader.readLine(BufferedReader.java:362) org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340) org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308) org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198) Thread 228 (SocketListener0-6): State: RUNNABLE Blocked count: 0 Waited count: 63 Stack: java.net.SocketInputStream.socketRead0(Native Method) java.net.SocketInputStream.read(SocketInputStream.java:129) org.mortbay.util.LineInput.fill(LineInput.java:469) org.mortbay.util.LineInput.fillLine(LineInput.java:547) org.mortbay.util.LineInput.readLineBuffer(LineInput.java:293) org.mortbay.util.LineInput.readLineBuffer(LineInput.java:277) org.mortbay.http.HttpRequest.readHeader(HttpRequest.java:238) org.mortbay.http.HttpConnection.readRequest(HttpConnection.java:861) org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:907) org.mortbay.http.HttpConnection.handle(HttpConnection.java:831) org.mortbay.http.SocketListener.handleConnection(SocketListener.java :244) org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357) org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534) Thread 20 ([EMAIL PROTECTED]): State: TIMED_WAITING Blocked count: 0 Waited count: 0 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.dfs.DFSClient$LeaseChecker.run(DFSClient.java:462) java.lang.Thread.run(Thread.java:595) Thread 16 (org.apache.hadoop.io.ObjectWritable Connection Culler): State: TIMED_WAITING Blocked count: 2 Waited count: 0 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.ipc.Client$ConnectionCuller.run(Client.java:388) Thread 15 (IPC Server handler 1 on 50050): State: TIMED_WAITING Blocked count: 2 Waited count: 732 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Server$Handler.run(Server.java:490) Thread 14 (IPC Server handler 0 on 50050): State: TIMED_WAITING Blocked count: 3 Waited count: 732 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Server$Handler.run(Server.java:490) Thread 13 (IPC Server listener on 50050): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.PollArrayWrapper.poll0(Native Method) sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100) sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) org.apache.hadoop.ipc.Server$Listener.run(Server.java:224) Thread 11 (Acceptor ServerSocket[addr= 0.0.0.0/0.0.0.0,port=0,localport=50060]): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: java.net.PlainSocketImpl.socketAccept(Native Method) java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384) java.net.ServerSocket.implAccept(ServerSocket.java:450) java.net.ServerSocket.accept(ServerSocket.java:421) org.mortbay.util.ThreadedServer.acceptSocket(ThreadedServer.java:432) org.mortbay.util.ThreadedServer$Acceptor.run(ThreadedServer.java:631) Thread 10 (SessionScavenger): State: TIMED_WAITING Blocked count: 0 Waited count: 0 Stack: java.lang.Thread.sleep(Native Method) org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run( AbstractSessionManager.java:587) Thread 9 (taskCleanup): State: WAITING Blocked count: 0 Waited count: 1 Waiting on [EMAIL PROTECTED] Stack: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:474) org.apache.hadoop.mapred.TaskTracker$BlockingQueue.take(TaskTracker.java :783) org.apache.hadoop.mapred.TaskTracker$1.run(TaskTracker.java:143) java.lang.Thread.run(Thread.java:595) Thread 4 (Signal Dispatcher): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: Thread 3 (Finalizer): State: WAITING Blocked count: 0 Waited count: 24 Waiting on [EMAIL PROTECTED] Stack: java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132) java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) Thread 2 (Reference Handler): State: WAITING Blocked count: 3 Waited count: 25 Waiting on [EMAIL PROTECTED] Stack: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:474) java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) Thread 1 (main): State: RUNNABLE Blocked count: 19 Waited count: 1499 Stack: sun.management.ThreadImpl.getThreadInfo0(Native Method) sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:144) sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:120) org.apache.hadoop.util.ReflectionUtils.printThreadInfo( ReflectionUtils.java:93) org.apache.hadoop.util.ReflectionUtils.logThreadInfo( ReflectionUtils.java:141) org.apache.hadoop.mapred.TaskTracker.markUnresponsiveTasks( TaskTracker.java:553) org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:449) org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:720) org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:1374)
