[ https://issues.apache.org/jira/browse/HADOOP-15335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yiran Wu updated HADOOP-15335: ------------------------------ Description: Print stack information and other info show in WebUI http://namenode:50070/stacks?contentionTracing=true {code:java} Thread 2 (Reference Handler): State: WAITING Blocked count: 8 Waited count: 5 Thread CpuTime: 5910000 Thread UserTime: 5754000 Thread allocatedBytes: 0 Waiting on java.lang.ref.Reference$Lock@4b3ed2f0 Blocked by -1 Stack: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) java.lang.ref.Reference.tryHandlePending(Reference.java:191) java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) Thread 1 (main): State: WAITING Blocked count: 4 Waited count: 2 Thread CpuTime: 2563937000 Thread UserTime: 2222977000 Thread allocatedBytes: 229115520 Waiting on org.apache.hadoop.ipc.ProtobufRpcEngine$Server@442a2e48 Blocked by -1 Stack: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.ipc.Server.join(Server.java:2498) org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.join(NameNodeRpcServer.java:442) org.apache.hadoop.hdfs.server.namenode.NameNode.join(NameNode.java:865) org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1573) --------------------------------------------------------- Locks info: --------------------------------------------------------- java.util.concurrent.locks.ReentrantReadWriteLock$FairSync@cd6c71a lockedBy Thread 43 (IPC Server handler 7 on 8020) Waiting thread num: 6 Stack: org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renameTo(FSNamesystem.java:3665) org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rename(NameNodeRpcServer.java:868) org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:583) org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2076) org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2072) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:422) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1803) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2072) java.lang.ref.ReferenceQueue$Lock@31bcf236 lockedBy UNKNOW java.lang.ref.Reference$Lock@4b3ed2f0 lockedBy UNKNOW org.apache.hadoop.ipc.ProtobufRpcEngine$Server@442a2e48 lockedBy UNKNOW {code} was: Print stack information and other info show in WebUI http://namenode:50070/stacks?contentionTracing=true ``` Thread 2 (Reference Handler): State: WAITING Blocked count: 8 Waited count: 5 Thread CpuTime: 5910000 Thread UserTime: 5754000 Thread allocatedBytes: 0 Waiting on java.lang.ref.Reference$Lock@4b3ed2f0 Blocked by -1 Stack: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) java.lang.ref.Reference.tryHandlePending(Reference.java:191) java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) Thread 1 (main): State: WAITING Blocked count: 4 Waited count: 2 Thread CpuTime: 2563937000 Thread UserTime: 2222977000 Thread allocatedBytes: 229115520 Waiting on org.apache.hadoop.ipc.ProtobufRpcEngine$Server@442a2e48 Blocked by -1 Stack: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.ipc.Server.join(Server.java:2498) org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.join(NameNodeRpcServer.java:442) org.apache.hadoop.hdfs.server.namenode.NameNode.join(NameNode.java:865) org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1573) --------------------------------------------------------- Locks info: --------------------------------------------------------- java.util.concurrent.locks.ReentrantReadWriteLock$FairSync@cd6c71a lockedBy Thread 43 (IPC Server handler 7 on 8020) Waiting thread num: 6 Stack: org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renameTo(FSNamesystem.java:3665) org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rename(NameNodeRpcServer.java:868) org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:583) org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2076) org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2072) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:422) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1803) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2072) java.lang.ref.ReferenceQueue$Lock@31bcf236 lockedBy UNKNOW java.lang.ref.Reference$Lock@4b3ed2f0 lockedBy UNKNOW org.apache.hadoop.ipc.ProtobufRpcEngine$Server@442a2e48 lockedBy UNKNOW ``` > Support xxxxxxx:xxx/stacks print lock info and more useful attribute of > thread info > ----------------------------------------------------------------------------------- > > Key: HADOOP-15335 > URL: https://issues.apache.org/jira/browse/HADOOP-15335 > Project: Hadoop Common > Issue Type: Improvement > Components: common > Affects Versions: 3.0.0 > Reporter: Yiran Wu > Priority: Major > Attachments: HADOOP-15335.001.patch, HADOOP-15335.002.patch > > > Print stack information and other info show in WebUI > http://namenode:50070/stacks?contentionTracing=true > {code:java} > Thread 2 (Reference Handler): > State: WAITING > Blocked count: 8 > Waited count: 5 > Thread CpuTime: 5910000 > Thread UserTime: 5754000 > Thread allocatedBytes: 0 > Waiting on java.lang.ref.Reference$Lock@4b3ed2f0 > Blocked by -1 > Stack: > java.lang.Object.wait(Native Method) > java.lang.Object.wait(Object.java:502) > java.lang.ref.Reference.tryHandlePending(Reference.java:191) > java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) > Thread 1 (main): > State: WAITING > Blocked count: 4 > Waited count: 2 > Thread CpuTime: 2563937000 > Thread UserTime: 2222977000 > Thread allocatedBytes: 229115520 > Waiting on org.apache.hadoop.ipc.ProtobufRpcEngine$Server@442a2e48 > Blocked by -1 > Stack: > java.lang.Object.wait(Native Method) > java.lang.Object.wait(Object.java:502) > org.apache.hadoop.ipc.Server.join(Server.java:2498) > > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.join(NameNodeRpcServer.java:442) > org.apache.hadoop.hdfs.server.namenode.NameNode.join(NameNode.java:865) > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1573) > --------------------------------------------------------- > Locks info: > --------------------------------------------------------- > java.util.concurrent.locks.ReentrantReadWriteLock$FairSync@cd6c71a lockedBy > Thread 43 (IPC Server handler 7 on 8020) > Waiting thread num: 6 > Stack: > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renameTo(FSNamesystem.java:3665) > > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rename(NameNodeRpcServer.java:868) > > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:583) > > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2076) > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2072) > java.security.AccessController.doPrivileged(Native Method) > javax.security.auth.Subject.doAs(Subject.java:422) > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1803) > org.apache.hadoop.ipc.Server$Handler.run(Server.java:2072) > java.lang.ref.ReferenceQueue$Lock@31bcf236 lockedBy UNKNOW > java.lang.ref.Reference$Lock@4b3ed2f0 lockedBy UNKNOW > org.apache.hadoop.ipc.ProtobufRpcEngine$Server@442a2e48 lockedBy UNKNOW > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org