Hi all I find that some if my DNs go to dead. and the datanode log shows as [1]:
I got the java.lang.OutOfMemoryError: Java heap space. I wander how this could come out. [1]: 1496 at file /home/hadoop/datadir/current/BP-471453121-172.16.250.16-1369298226760/current/finalized/subdir56/subdir46/blk_8143366700047777895 2013-06-07 13:49:35,526 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted block BP-471453121-172.16.250.16-1369298226760 blk_7448905973553274685_691506 at file /home/hadoop/datadir/current/BP-471453121-172.16.250.16-1369298226760/current/finalized/subdir56/subdir46/blk_7448905973553274685 2013-06-07 13:49:35,526 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted block BP-471453121-172.16.250.16-1369298226760 blk_-1754381991139616539_691508 at file /home/hadoop/datadir/current/BP-471453121-172.16.250.16-1369298226760/current/finalized/subdir56/subdir46/blk_-1754381991139616539 2013-06-07 13:49:35,527 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted block BP-471453121-172.16.250.16-1369298226760 blk_-3379745807160766937_691492 at file /home/hadoop/datadir/current/BP-471453121-172.16.250.16-1369298226760/current/finalized/subdir56/subdir46/blk_-3379745807160766937 2013-06-07 13:51:44,231 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in offerService java.io.IOException: com.google.protobuf.ServiceException: java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.ipc.ProtobufHelper.getRemoteException(ProtobufHelper.java:47) at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReport(DatanodeProtocolClientSideTranslatorPB.java:203) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport(BPServiceActor.java:399) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:551) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:674) at java.lang.Thread.run(Thread.java:662) Caused by: com.google.protobuf.ServiceException: java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:212) at $Proxy10.blockReport(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83) at $Proxy10.blockReport(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReport(DatanodeProtocolClientSideTranslatorPB.java:201) ... 4 more Caused by: java.lang.OutOfMemoryError: Java heap space 2013-06-07 13:51:46,983 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected exception in block pool Block pool BP-471453121-172.16.250.16-1369298226760 (storage id DS-1482330176-172.16.250.19-50010-1369298284371) service to wxossetl2/ 172.16.250.16:8020 java.lang.OutOfMemoryError: Java heap space 2013-06-07 13:51:46,983 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool BP-471453121-172.16.250.16-1369298226760 (storage id DS-1482330176-172.16.250.19-50010-1369298284371) service to wxossetl2/ 172.16.250.16:8020 2013-06-07 13:51:47,090 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool BP-471453121-172.16.250.16-1369298226760 (storage id DS-1482330176-172.16.250.19-50010-1369298284371) 2013-06-07 13:51:47,090 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Removed bpid=BP-471453121-172.16.250.16-1369298226760 from blockPoolScannerMap 2013-06-07 13:51:47,090 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Removing block pool BP-471453121-172.16.250.16-1369298226760 2013-06-07 13:51:49,091 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode 2013-06-07 13:51:49,098 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 0 2013-06-07 13:51:49,101 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down DataNode at wxossetl5/172.16.250.19
