Hey guys, I know it's 5PM on a Friday, but we just saw one of our big cluster's namenode's deadlock.
This is 0.19.1; does this ring a bell for anyone? I haven't had any time to
start going through source code, but I figured I'd send out a SOS in case if
this looked familiar.
We had restarted this cluster a few hours ago and made the following changes:
1) Increased the number of datanode handlers from 10 to 40.
2) Increased ipc.server.listen.queue.size from 128 to 256.
If nothing else, I figure a deadlocked NN might be interesting to devs...
Brian
2010-05-14 17:11:30
Full thread dump Java HotSpot(TM) 64-Bit Server VM (11.2-b01 mixed mode):
"IPC Server handler 39 on 9000" daemon prio=10 tid=0x00002aaafc181400 nid=0x4cd
waiting for monitor entry [0x0000000045962000..0x0000000045962d90]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
- waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 38 on 9000" daemon prio=10 tid=0x00002aaafc17f800 nid=0x4cc
waiting for monitor entry [0x0000000045861000..0x0000000045861d10]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getStats(FSNamesystem.java:3326)
- waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getStats(NameNode.java:505)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 37 on 9000" daemon prio=10 tid=0x00002aaafc17e000 nid=0x4cb
waiting for monitor entry [0x000000004575f000..0x0000000045760a90]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 36 on 9000" daemon prio=10 tid=0x00002aaafc17c400 nid=0x4ca
waiting for monitor entry [0x000000004565e000..0x000000004565fa10]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:3281)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:649)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 35 on 9000" daemon prio=10 tid=0x00002aaafc17a800 nid=0x4c9
waiting for monitor entry [0x000000004555e000..0x000000004555eb90]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 34 on 9000" daemon prio=10 tid=0x00002aaafc178c00 nid=0x4c8
waiting for monitor entry [0x000000004545d000..0x000000004545db10]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 33 on 9000" daemon prio=10 tid=0x00002aaafc177400 nid=0x4c7
waiting for monitor entry [0x000000004535c000..0x000000004535cc90]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
- waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 32 on 9000" daemon prio=10 tid=0x00002aaafc175800 nid=0x4c6
waiting for monitor entry [0x000000004525a000..0x000000004525bc10]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 31 on 9000" daemon prio=10 tid=0x00002aaafc173c00 nid=0x4c5
waiting for monitor entry [0x000000004515a000..0x000000004515ad90]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:3281)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:649)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 30 on 9000" daemon prio=10 tid=0x00002aaafc172400 nid=0x4c4
waiting for monitor entry [0x0000000045059000..0x0000000045059d10]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
- waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 29 on 9000" daemon prio=10 tid=0x00002aaafc171000 nid=0x4c3
waiting for monitor entry [0x0000000044f57000..0x0000000044f58a90]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
- waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 28 on 9000" daemon prio=10 tid=0x00002aaafc16f400 nid=0x4c2
waiting for monitor entry [0x0000000044e56000..0x0000000044e57a10]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 27 on 9000" daemon prio=10 tid=0x00002aaafc16d800 nid=0x4c1
waiting for monitor entry [0x0000000044d56000..0x0000000044d56b90]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 26 on 9000" daemon prio=10 tid=0x00002aaafc16bc00 nid=0x4c0
waiting for monitor entry [0x0000000044c55000..0x0000000044c55b10]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 25 on 9000" daemon prio=10 tid=0x00002aaafc16a400 nid=0x4bf
waiting for monitor entry [0x0000000044b54000..0x0000000044b54c90]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
- waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 24 on 9000" daemon prio=10 tid=0x00002aaafc168800 nid=0x4be
waiting for monitor entry [0x0000000044a53000..0x0000000044a53c10]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
- waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 23 on 9000" daemon prio=10 tid=0x00002aaafc166c00 nid=0x4bd
waiting for monitor entry [0x0000000044952000..0x0000000044952d90]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 22 on 9000" daemon prio=10 tid=0x00002aaafc165400 nid=0x4bc
waiting for monitor entry [0x0000000044851000..0x0000000044851d10]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 21 on 9000" daemon prio=10 tid=0x00002aaafc164000 nid=0x4bb
waiting for monitor entry [0x0000000044750000..0x0000000044750a90]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 20 on 9000" daemon prio=10 tid=0x00002aaafc162400 nid=0x4ba
waiting for monitor entry [0x000000004464f000..0x000000004464fa10]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 19 on 9000" daemon prio=10 tid=0x00002aaafc488400 nid=0x4b9
waiting for monitor entry [0x000000004454e000..0x000000004454eb90]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 18 on 9000" daemon prio=10 tid=0x00002aaafc486800 nid=0x4b8
waiting for monitor entry [0x000000004444d000..0x000000004444db10]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.nextGenerationStampForBlock(FSNamesystem.java:4548)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.nextGenerationStamp(NameNode.java:402)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 17 on 9000" daemon prio=10 tid=0x00002aaafc484c00 nid=0x4b7
waiting for monitor entry [0x000000004434c000..0x000000004434cc90]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.nextGenerationStampForBlock(FSNamesystem.java:4548)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.nextGenerationStamp(NameNode.java:402)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 16 on 9000" daemon prio=10 tid=0x00002aaafc483000 nid=0x4b6
waiting for monitor entry [0x000000004424a000..0x000000004424bc10]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 15 on 9000" daemon prio=10 tid=0x00002aaafc481800 nid=0x4b5
waiting for monitor entry [0x000000004414a000..0x000000004414ad90]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 14 on 9000" daemon prio=10 tid=0x00002aaafc47fc00 nid=0x4b4
waiting for monitor entry [0x0000000044049000..0x0000000044049d10]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
- waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 13 on 9000" daemon prio=10 tid=0x00002aaafc47e000 nid=0x4b3
waiting for monitor entry [0x0000000043f48000..0x0000000043f48a90]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
- waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 12 on 9000" daemon prio=10 tid=0x00002aaafc47c400 nid=0x4b2
waiting for monitor entry [0x0000000043e46000..0x0000000043e47a10]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 11 on 9000" daemon prio=10 tid=0x00002aaafc47b400 nid=0x4b1
waiting for monitor entry [0x0000000043d45000..0x0000000043d46b90]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 10 on 9000" daemon prio=10 tid=0x00002aaafc479800 nid=0x4b0
waiting for monitor entry [0x0000000043c45000..0x0000000043c45b10]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 9 on 9000" daemon prio=10 tid=0x00002aaafc477c00 nid=0x4af
waiting for monitor entry [0x0000000043b43000..0x0000000043b44c90]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 8 on 9000" daemon prio=10 tid=0x00002aaafc476400 nid=0x4ae
waiting for monitor entry [0x0000000043a42000..0x0000000043a43c10]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 7 on 9000" daemon prio=10 tid=0x00002aaafc474c00 nid=0x4ad
waiting for monitor entry [0x0000000043942000..0x0000000043942d90]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 6 on 9000" daemon prio=10 tid=0x00002aaafc473000 nid=0x4ac
waiting for monitor entry [0x0000000043841000..0x0000000043841d10]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 5 on 9000" daemon prio=10 tid=0x00002aaafc471400 nid=0x4ab
waiting for monitor entry [0x0000000043740000..0x0000000043740a90]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 4 on 9000" daemon prio=10 tid=0x00002aaafc121800 nid=0x4aa
waiting for monitor entry [0x000000004363f000..0x000000004363fa10]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 3 on 9000" daemon prio=10 tid=0x00002aaafc11fc00 nid=0x4a9
waiting for monitor entry [0x000000004353d000..0x000000004353eb90]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 2 on 9000" daemon prio=10 tid=0x00002aaafc11e400 nid=0x4a8
waiting for monitor entry [0x000000004343c000..0x000000004343db10]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 1 on 9000" daemon prio=10 tid=0x00002aaafc467800 nid=0x4a7
waiting for monitor entry [0x000000004333b000..0x000000004333cc90]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server handler 0 on 9000" daemon prio=10 tid=0x00002aaafc466400 nid=0x4a6
waiting for monitor entry [0x000000004323b000..0x000000004323bc10]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
"IPC Server listener on 9000" daemon prio=10 tid=0x00002aaafc12fc00 nid=0x4a5
waiting on condition [0x000000004313a000..0x000000004313ad90]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaab3b8fd70> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:254)
at org.apache.hadoop.ipc.Server$Connection.processData(Server.java:850)
at
org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:814)
at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:400)
at org.apache.hadoop.ipc.Server$Listener.run(Server.java:309)
"IPC Server Responder" daemon prio=10 tid=0x00002aaafc0bbc00 nid=0x4a4 runnable
[0x0000000043039000..0x0000000043039d10]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00002aaab3b90e80> (a sun.nio.ch.Util$1)
- locked <0x00002aaab3b90e68> (a java.util.Collections$UnmodifiableSet)
- locked <0x00002aaab3b90b08> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.apache.hadoop.ipc.Server$Responder.run(Server.java:458)
"SocketListener0-1" prio=10 tid=0x00002aaafc0ba400 nid=0x4a3 in Object.wait()
[0x0000000042f38000..0x0000000042f38a90]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaadf36cb00> (a
org.mortbay.util.ThreadPool$PoolThread)
at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:522)
- locked <0x00002aaadf36cb00> (a org.mortbay.util.ThreadPool$PoolThread)
"SocketListener0-0" prio=10 tid=0x00002aaafc0b4c00 nid=0x4a2 in Object.wait()
[0x0000000042e37000..0x0000000042e37a10]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaadf36c820> (a
org.mortbay.util.ThreadPool$PoolThread)
at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:522)
- locked <0x00002aaadf36c820> (a org.mortbay.util.ThreadPool$PoolThread)
"Acceptor ServerSocket[addr=proxy-1/169.228.130.63,port=0,localport=50070]"
prio=10 tid=0x00002aaafc0c8400 nid=0x4a1 runnable
[0x0000000042d36000..0x0000000042d36b90]
java.lang.Thread.State: RUNNABLE
at java.lang.Object.<init>(Object.java:20)
at java.net.SocketImpl.<init>(SocketImpl.java:27)
at java.net.PlainSocketImpl.<init>(PlainSocketImpl.java:90)
at java.net.SocksSocketImpl.<init>(SocksSocketImpl.java:33)
at java.net.Socket.setImpl(Socket.java:434)
at java.net.ServerSocket.implAccept(ServerSocket.java:445)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.mortbay.util.ThreadedServer.acceptSocket(ThreadedServer.java:432)
at org.mortbay.util.ThreadedServer$Acceptor.run(ThreadedServer.java:631)
"SessionScavenger" daemon prio=10 tid=0x00002aaafc0c1000 nid=0x4a0 waiting on
condition [0x0000000042c35000..0x0000000042c35b10]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)
"SessionScavenger" daemon prio=10 tid=0x00002aaafc0fa000 nid=0x49f waiting on
condition [0x0000000040b2a000..0x0000000040b2ac90]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)
"SessionScavenger" daemon prio=10 tid=0x00002aaafc139800 nid=0x49a waiting on
condition [0x00000000408b7000..0x00000000408b7c10]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)
"org.apache.hadoop.hdfs.server.namenode.decommissionmanager$moni...@1b0deb5f"
daemon prio=10 tid=0x00002aaafc0a2800 nid=0x496 waiting for monitor entry
[0x0000000042b34000..0x0000000042b34d90]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:61)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at java.lang.Thread.run(Thread.java:619)
"org.apache.hadoop.hdfs.server.namenode.fsnamesystem$replicationmoni...@784a7df6"
daemon prio=10 tid=0x00002aaafc090c00 nid=0x495 waiting for monitor entry
[0x00000000411f8000..0x00000000411f8d10]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.computeReplicationWork(FSNamesystem.java:2410)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.computeDatanodeWork(FSNamesystem.java:2376)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2333)
at java.lang.Thread.run(Thread.java:619)
"org.apache.hadoop.hdfs.server.namenode.leasemanager$moni...@d24111a" daemon
prio=10 tid=0x00002aaafc013400 nid=0x494 waiting for monitor entry
[0x00000000410f7000..0x00000000410f7a90]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:349)
- waiting to lock <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at java.lang.Thread.run(Thread.java:619)
"org.apache.hadoop.hdfs.server.namenode.fsnamesystem$heartbeatmoni...@1c778255"
daemon prio=10 tid=0x00002aaafc012c00 nid=0x493 runnable
[0x00000000413da000..0x00000000413daa10]
java.lang.Thread.State: RUNNABLE
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeStoredBlock(FSNamesystem.java:3236)
- locked <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeDatanode(FSNamesystem.java:2695)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.heartbeatCheck(FSNamesystem.java:2785)
- locked <0x00002aaab3659dd8> (a java.util.TreeMap)
- locked <0x00002aaab3653848> (a java.util.ArrayList)
- locked <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2312)
at java.lang.Thread.run(Thread.java:619)
"org.apache.hadoop.hdfs.server.namenode.pendingreplicationblocks$pendingreplicationmoni...@44676e3f"
daemon prio=10 tid=0x00002aaafc012800 nid=0x492 waiting on condition
[0x0000000040d16000..0x0000000040d16b90]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
at java.lang.Thread.run(Thread.java:619)
"Timer thread for monitoring dfs" daemon prio=10 tid=0x00002aaafc03d800
nid=0x428 waiting for monitor entry [0x0000000040e7e000..0x0000000040e7ed10]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getStats(FSNamesystem.java:3326)
- waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getCapacityTotal(FSNamesystem.java:3335)
at
org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics.doUpdates(FSNamesystemMetrics.java:90)
- locked <0x00002aaab3b3c580> (a
org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics)
at
org.apache.hadoop.metrics.spi.AbstractMetricsContext.timerEvent(AbstractMetricsContext.java:284)
at
org.apache.hadoop.metrics.spi.AbstractMetricsContext.access$000(AbstractMetricsContext.java:50)
at
org.apache.hadoop.metrics.spi.AbstractMetricsContext$1.run(AbstractMetricsContext.java:249)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
"Low Memory Detector" daemon prio=10 tid=0x000000005c306c00 nid=0x426 runnable
[0x0000000000000000..0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread1" daemon prio=10 tid=0x000000005c304000 nid=0x425 waiting on
condition [0x0000000000000000..0x0000000042830420]
java.lang.Thread.State: RUNNABLE
"CompilerThread0" daemon prio=10 tid=0x000000005c300000 nid=0x424 waiting on
condition [0x0000000000000000..0x000000004272f3a0]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x000000005c2fe400 nid=0x423 waiting on
condition [0x0000000000000000..0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x000000005c2da800 nid=0x422 in Object.wait()
[0x000000004252e000..0x000000004252ec10]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab38ec320> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
- locked <0x00002aaab38ec320> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x000000005c2d8c00 nid=0x421 in
Object.wait() [0x000000004177e000..0x000000004177ed90]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab38e9ec0> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x00002aaab38e9ec0> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x000000005c268400 nid=0x417 in Object.wait()
[0x0000000040494000..0x0000000040494ed0]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab3b3db50> (a org.apache.hadoop.ipc.RPC$Server)
at java.lang.Object.wait(Object.java:485)
at org.apache.hadoop.ipc.Server.join(Server.java:1023)
- locked <0x00002aaab3b3db50> (a org.apache.hadoop.ipc.RPC$Server)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.join(NameNode.java:221)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:870)
"VM Thread" prio=10 tid=0x000000005c2d3400 nid=0x420 runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x000000005c273000 nid=0x418
runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x000000005c274800 nid=0x419
runnable
"GC task thread#2 (ParallelGC)" prio=10 tid=0x000000005c276400 nid=0x41a
runnable
"GC task thread#3 (ParallelGC)" prio=10 tid=0x000000005c277c00 nid=0x41b
runnable
"GC task thread#4 (ParallelGC)" prio=10 tid=0x000000005c279400 nid=0x41c
runnable
"GC task thread#5 (ParallelGC)" prio=10 tid=0x000000005c27b000 nid=0x41d
runnable
"GC task thread#6 (ParallelGC)" prio=10 tid=0x000000005c27c800 nid=0x41e
runnable
"GC task thread#7 (ParallelGC)" prio=10 tid=0x000000005c27e000 nid=0x41f
runnable
"VM Periodic Task Thread" prio=10 tid=0x000000005c308c00 nid=0x427 waiting on
condition
JNI global references: 984
Heap
PSYoungGen total 203712K, used 107631K [0x00002aaadd0b0000,
0x00002aaaf1e00000, 0x00002aaaf1e00000)
eden space 124096K, 86% used
[0x00002aaadd0b0000,0x00002aaae39cbe58,0x00002aaae49e0000)
from space 79616K, 0% used
[0x00002aaae49e0000,0x00002aaae49e0000,0x00002aaae97a0000)
to space 113728K, 0% used
[0x00002aaaeaef0000,0x00002aaaeaef0000,0x00002aaaf1e00000)
PSOldGen total 682688K, used 682687K [0x00002aaab3600000,
0x00002aaadd0b0000, 0x00002aaadd0b0000)
object space 682688K, 99% used
[0x00002aaab3600000,0x00002aaadd0affe8,0x00002aaadd0b0000)
PSPermGen total 21248K, used 16689K [0x00002aaaae200000,
0x00002aaaaf6c0000, 0x00002aaab3600000)
object space 21248K, 78% used
[0x00002aaaae200000,0x00002aaaaf24c660,0x00002aaaaf6c0000)
smime.p7s
Description: S/MIME cryptographic signature
