Just wanted to chime in and say that I have experienced this often
too. The task eventually completes, but after an inordinate amount of
time. The last lone task gets stuck in the initialiazation stage.
It's usually faster to restart the job.
On 11/17/06, Johan Oskarsson <[EMAIL PROTECTED]> wrote:
Have anyone experienced this?
When the cluster is busy sometimes one map task of a job gets stuck.
An attempt is then made to kill the task, but this fails.
This shows up in the web interface:
task_0028_m_000534_0 node17.herd1 RUNNING 0.00% 10-Nov-2006 12:21:12
10-Nov-2006 12:22:19 (1mins, 6sec)
Task failed to report status for 604 seconds. Killing.
And that's the way it stays. Only way to get rid of it / rescue the job is
to restart the tasktracker.
This is what I found in the log a few times.
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(
SocketChannelImpl.java:125)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:294)
at org.apache.hadoop.ipc.SocketChannelOutputStream.flushBuffer(
SocketChannelOutputStream.java:108)
at org.apache.hadoop.ipc.SocketChannelOutputStream.write(
SocketChannelOutputStream.java:89)
at java.io.BufferedOutputStream.flushBuffer(
BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:532)
I've filed an issue about this at:
http://issues.apache.org/jira/browse/HADOOP-707
The open file limit is at 65000, so shouldn't be a problem.
All nodes are on the same network switch.
Full thread dump:
2006-11-17 15:08:14,707 INFO org.apache.hadoop.mapred.TaskTracker:
task_0133_m_000948_0: Task failed to report status for 608 seconds. Killing.
2006-11-17 15:08:14,722 INFO org.apache.hadoop.mapred.TaskTracker: Process
Thread Dump: lost task
65 active threads
Thread 46827 (IPC Client connection to master1.herd1/10.0.5.10:9001):
State: WAITING
Blocked count: 1
Waited count: 2
Waiting on [EMAIL PROTECTED]
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:474)
org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:195)
org.apache.hadoop.ipc.Client$Connection.run(Client.java:234)
Thread 46826 (btpool0-893):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46825 (btpool0-892):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46822 (btpool0-891):
State: RUNNABLE
Blocked count: 0
Waited count: 1
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46811 (btpool0-890):
State: RUNNABLE
Blocked count: 3
Waited count: 1
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46810 (btpool0-889):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46803 (btpool0-888):
State: RUNNABLE
Blocked count: 0
Waited count: 8
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46408 (btpool0-887):
State: RUNNABLE
Blocked count: 0
Waited count: 41
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46402 (btpool0-886):
State: RUNNABLE
Blocked count: 0
Waited count: 26
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46396 (btpool0-885):
State: RUNNABLE
Blocked count: 4
Waited count: 48
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
org.apache.hadoop.fs.LocalFileSystem$LocalFSFileInputStream.read(
LocalFileSystem.java:113)
org.apache.hadoop.fs.FSDataInputStream$Checker.read(
FSDataInputStream.java:91)
org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(
FSDataInputStream.java:189)
java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
java.io.DataInputStream.read(DataInputStream.java:80)
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(
TaskTracker.java:1411)
javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:445)
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:356)
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:226)
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:627)
org.mortbay.jetty.handler.ContextHandlerCollection.handle(
ContextHandlerCollection.java:149)
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:141)
org.mortbay.jetty.Server.handle(Server.java:269)
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:430)
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(
HttpConnection.java:687)
Thread 46345 (btpool0-883):
State: TIMED_WAITING
Blocked count: 0
Waited count: 26
Stack:
java.lang.Object.wait(Native Method)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:469)
Thread 46333 (btpool0-881):
State: RUNNABLE
Blocked count: 0
Waited count: 42
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46305 (btpool0-880):
State: RUNNABLE
Blocked count: 0
Waited count: 43
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46304 (btpool0-879):
State: RUNNABLE
Blocked count: 0
Waited count: 37
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46295 (btpool0-878):
State: RUNNABLE
Blocked count: 1
Waited count: 36
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46253 (btpool0-877):
State: RUNNABLE
Blocked count: 1
Waited count: 48
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46252 (btpool0-876):
State: RUNNABLE
Blocked count: 0
Waited count: 46
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46251 (btpool0-875):
State: RUNNABLE
Blocked count: 0
Waited count: 51
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46242 (btpool0-874):
State: RUNNABLE
Blocked count: 0
Waited count: 43
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 45039 (Thread-28307):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
Thread 45038 (process reaper):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.lang.UNIXProcess.waitForProcessExit(Native Method)
java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
Thread 45031 (Thread-28302):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
Thread 45030 (process reaper):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.lang.UNIXProcess.waitForProcessExit(Native Method)
java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
Thread 45026 (Thread-28299):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
Thread 45025 (process reaper):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.lang.UNIXProcess.waitForProcessExit(Native Method)
java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
Thread 44998 (Thread-28283):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
Thread 44997 (process reaper):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.lang.UNIXProcess.waitForProcessExit(Native Method)
java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
Thread 44987 (Thread-28277):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
Thread 44986 (process reaper):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.lang.UNIXProcess.waitForProcessExit(Native Method)
java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
Thread 44905 (btpool0-872):
State: TIMED_WAITING
Blocked count: 2
Waited count: 87
Stack:
java.lang.Object.wait(Native Method)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:469)
Thread 44903 (btpool0-870):
State: TIMED_WAITING
Blocked count: 1
Waited count: 103
Stack:
java.lang.Object.wait(Native Method)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:469)
Thread 44891 (btpool0-865):
State: RUNNABLE
Blocked count: 5
Waited count: 55
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 44888 (btpool0-863):
State: RUNNABLE
Blocked count: 7
Waited count: 135
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 44887 (btpool0-862):
State: RUNNABLE
Blocked count: 4
Waited count: 131
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 44882 (btpool0-857):
State: RUNNABLE
Blocked count: 9
Waited count: 138
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 44881 (btpool0-856):
State: RUNNABLE
Blocked count: 3
Waited count: 127
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 44841 (btpool0-851):
State: RUNNABLE
Blocked count: 6
Waited count: 127
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 44752 (btpool0-849):
State: RUNNABLE
Blocked count: 7
Waited count: 134
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 43631 (Thread-27401):
State: RUNNABLE
Blocked count: 114
Waited count: 1730
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
Thread 43610 (Thread-27384):
State: RUNNABLE
Blocked count: 116
Waited count: 1687
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
Thread 43591 (Thread-27369):
State: RUNNABLE
Blocked count: 79
Waited count: 1690
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
Thread 43583 (Thread-27361):
State: RUNNABLE
Blocked count: 109
Waited count: 1690
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
Thread 43575 (Thread-27353):
State: RUNNABLE
Blocked count: 128
Waited count: 1726
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
Thread 43052 (btpool0-844):
State: RUNNABLE
Blocked count: 6
Waited count: 217
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 43049 (btpool0-842):
State: RUNNABLE
Blocked count: 4
Waited count: 213
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 43042 (btpool0-835):
State: RUNNABLE
Blocked count: 12
Waited count: 229
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
org.apache.hadoop.fs.LocalFileSystem$LocalFSFileInputStream.read(
LocalFileSystem.java:113)
org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(
FSDataInputStream.java:189)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
java.io.DataInputStream.readFully(DataInputStream.java:176)
java.io.DataInputStream.readFully(DataInputStream.java:152)
org.apache.hadoop.fs.FSDataInputStream$Checker.<init>(
FSDataInputStream.java:60)
org.apache.hadoop.fs.FSDataInputStream.<init>(FSDataInputStream.java
:279)
org.apache.hadoop.fs.FileSystem.open(FileSystem.java:156)
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(
TaskTracker.java:1409)
javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:445)
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:356)
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:226)
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:627)
org.mortbay.jetty.handler.ContextHandlerCollection.handle(
ContextHandlerCollection.java:149)
Thread 40895 (btpool0-803):
State: RUNNABLE
Blocked count: 3
Waited count: 361
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 40380 (btpool0-790):
State: RUNNABLE
Blocked count: 7
Waited count: 391
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
org.apache.hadoop.fs.LocalFileSystem$LocalFSFileInputStream.read(
LocalFileSystem.java:113)
org.apache.hadoop.fs.FSDataInputStream$Checker.read(
FSDataInputStream.java:91)
org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(
FSDataInputStream.java:189)
java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
java.io.DataInputStream.read(DataInputStream.java:80)
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(
TaskTracker.java:1411)
javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:445)
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:356)
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:226)
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:627)
org.mortbay.jetty.handler.ContextHandlerCollection.handle(
ContextHandlerCollection.java:149)
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:141)
org.mortbay.jetty.Server.handle(Server.java:269)
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:430)
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(
HttpConnection.java:687)
Thread 39121 (btpool0-784):
State: RUNNABLE
Blocked count: 16
Waited count: 439
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 39112 (btpool0-775):
State: RUNNABLE
Blocked count: 13
Waited count: 396
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 28 ([EMAIL PROTECTED]):
State: TIMED_WAITING
Blocked count: 0
Waited count: 0
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.dfs.DFSClient$LeaseChecker.run(DFSClient.java:462)
java.lang.Thread.run(Thread.java:595)
Thread 19 (org.apache.hadoop.io.ObjectWritable Connection Culler):
State: TIMED_WAITING
Blocked count: 154
Waited count: 0
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.ipc.Client$ConnectionCuller.run(Client.java:388)
Thread 18 (IPC Server handler 4 on 50050):
State: TIMED_WAITING
Blocked count: 1769
Waited count: 51161
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:486)
Thread 17 (IPC Server handler 3 on 50050):
State: TIMED_WAITING
Blocked count: 1673
Waited count: 51202
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:486)
Thread 16 (IPC Server handler 2 on 50050):
State: TIMED_WAITING
Blocked count: 1685
Waited count: 51188
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:486)
Thread 15 (IPC Server handler 1 on 50050):
State: TIMED_WAITING
Blocked count: 1694
Waited count: 51183
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:486)
Thread 14 (IPC Server handler 0 on 50050):
State: TIMED_WAITING
Blocked count: 1913
Waited count: 51137
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:486)
Thread 13 (IPC Server listener on 50050):
State: RUNNABLE
Blocked count: 634
Waited count: 0
Stack:
sun.nio.ch.PollArrayWrapper.poll0(Native Method)
sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100)
sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener.run(Server.java:224)
Thread 12 (btpool0-1 - Acceptor0 SocketConnector @ 0.0.0.0:50060):
State: RUNNABLE
Blocked count: 85
Waited count: 1517
Stack:
java.net.PlainSocketImpl.socketAccept(Native Method)
java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
java.net.ServerSocket.implAccept(ServerSocket.java:450)
java.net.ServerSocket.accept(ServerSocket.java:421)
org.mortbay.jetty.bio.SocketConnector.accept(SocketConnector.java:97)
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java
:630)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 11 (btpool0-0 - Invalidator - /):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1
Stack:
java.lang.Thread.sleep(Native Method)
org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(
AbstractSessionManager.java:933)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 9 (taskCleanup):
State: WAITING
Blocked count: 457
Waited count: 118
Waiting on [EMAIL PROTECTED]
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:474)
org.apache.hadoop.mapred.TaskTracker$BlockingQueue.take(TaskTracker.java
:783)
org.apache.hadoop.mapred.TaskTracker$1.run(TaskTracker.java:143)
java.lang.Thread.run(Thread.java:595)
Thread 4 (Signal Dispatcher):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
Thread 3 (Finalizer):
State: WAITING
Blocked count: 99047
Waited count: 39648
Waiting on [EMAIL PROTECTED]
Stack:
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Thread 2 (Reference Handler):
State: WAITING
Blocked count: 102986
Waited count: 3131
Waiting on [EMAIL PROTECTED]
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:474)
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
Thread 1 (main):
State: RUNNABLE
Blocked count: 1538
Waited count: 107465
Stack:
sun.management.ThreadImpl.getThreadInfo0(Native Method)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:144)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:120)
org.apache.hadoop.util.ReflectionUtils.printThreadInfo(
ReflectionUtils.java:93)
org.apache.hadoop.util.ReflectionUtils.logThreadInfo(
ReflectionUtils.java:141)
org.apache.hadoop.mapred.TaskTracker.markUnresponsiveTasks(
TaskTracker.java:553)
org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:449)
org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:720)
org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:1374)
/Johan
