Hi, after a few rounds of spidering with nutch (2.2.1) hbase (0.90.6) is constantly crashing with this error message below.
I tried quite a lot already to change but no success. Any idea what I could try to adjust in order to make hbase work stable running on hadoop? Further below all the WARN/ERROR/FATAL logs from the most recent NUTCH spider I did which ended like: 0/50 spinwaiting/active, 49429 pages, 1429 errors, 26.6 0 pages/s, 15782 0 kb/s, 2500 URLs in 397 queues Aborting with 50 hung threads. Please keep in mind that hbase/nutch is working for a couple of rounds but then crashes suddently. =============== 2014-01-02 14:03:51,493 INFO org.apache.hadoop.hbase.master.HMaster: Stopping infoServer 2014-01-02 14:03:51,493 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server Responder 2014-01-02 14:03:51,493 INFO org.apache.hadoop.hbase.master.LogCleaner: master-localhost:60000.oldLogCleaner exiting 2014-01-02 14:03:51,494 INFO org.mortbay.log: Stopped [email protected]:60010 2014-01-02 14:03:51,609 DEBUG org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@329b5cdd 2014-01-02 14:03:51,609 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: The connection to hconnection-0x14352e9bb820003 has been closed. 2014-01-02 14:03:51,609 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Closed zookeeper sessionid=0x14352e9bb820003 2014-01-02 14:03:51,614 INFO org.apache.zookeeper.ZooKeeper: Session: 0x14352e9bb820003 closed 2014-01-02 14:03:51,614 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: The connection to null has been closed. 2014-01-02 14:03:51,614 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2014-01-02 14:03:51,614 INFO org.apache.hadoop.hbase.master.AssignmentManager$TimeoutMonitor: localhost:60000.timeoutMonitor exiting 2014-01-02 14:03:51,620 INFO org.apache.zookeeper.ZooKeeper: Session: 0x14352e9bb820000 closed 2014-01-02 14:03:51,620 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2014-01-02 14:03:51,620 INFO org.apache.hadoop.hbase.master.HMaster: HMaster main thread exiting 2014-01-02 14:03:51,642 ERROR org.apache.hadoop.hdfs.DFSClient: Failed to close file /hbase/3_webpage/9b4a1a1b189a78ef83d7b7c00ca791d0/recovered.edits/0000000000000007585.temp org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /hbase/3_webpage/9b4a1a1b189a78ef83d7b7c00ca791d0/recovered.edits/0000000000000007585.temp could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1920) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:783) at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426) at org.apache.hadoop.ipc.Client.call(Client.java:1113) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) at com.sun.proxy.$Proxy6.addBlock(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62) at com.sun.proxy.$Proxy6.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3720) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3580) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2783) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3023) 2014-01-02 14:03:51,655 ERROR org.apache.hadoop.hdfs.DFSClient: Failed to close file /hbase/3_webpage/5d7f60dbcb95c4b08e9a5c09a0ed4808/recovered.edits/0000000000000007587.temp org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /hbase/3_webpage/5d7f60dbcb95c4b08e9a5c09a0ed4808/recovered.edits/0000000000000007587.temp could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1920) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:783) at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426) at org.apache.hadoop.ipc.Client.call(Client.java:1113) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) at com.sun.proxy.$Proxy6.addBlock(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62) at com.sun.proxy.$Proxy6.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3720) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3580) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2783) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3023) 2014-01-02 14:03:51,656 ERROR org.apache.hadoop.hdfs.DFSClient: Failed to close file /hbase/3_webpage/2de53e798b85e736fa5dd86431746732/recovered.edits/0000000000000007586.temp org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /hbase/3_webpage/2de53e798b85e736fa5dd86431746732/recovered.edits/0000000000000007586.temp could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1920) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:783) at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426) at org.apache.hadoop.ipc.Client.call(Client.java:1113) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) at com.sun.proxy.$Proxy6.addBlock(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62) at com.sun.proxy.$Proxy6.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3720) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3580) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2783) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3023) 2014-01-02 15:29:08,328 WARN org.apache.hadoop.hbase.util.FSUtils: Running on HDFS without append enabled may result in data loss 2014-01-02 15:29:08,334 WARN org.apache.hadoop.hbase.util.FSUtils: Running on HDFS without append enabled may result in data loss 2014-01-02 15:29:08,340 WARN org.apache.hadoop.hbase.util.FSUtils: Running on HDFS without append enabled may result in data loss 2014-01-02 15:29:08,345 WARN org.apache.hadoop.hbase.util.FSUtils: Running on HDFS without append enabled may result in data loss 2014-01-02 15:29:08,352 WARN org.apache.hadoop.hbase.util.FSUtils: Running on HDFS without append enabled may result in data loss 2014-01-02 15:29:08,370 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /hbase/.META./1028785192/recovered.edits/0000000000000005582.temp could only be replicated to 0 nodes, instead of 1 2014-01-02 15:29:08,370 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for null bad datanode[0] nodes == null 2014-01-02 15:29:08,370 WARN org.apache.hadoop.hdfs.DFSClient: Could not get block locations. Source file "/hbase/.META./1028785192/recovered.edits/0000000000000005582.temp" - Aborting... 2014-01-02 15:29:08,122 ERROR org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Error in log splitting write thread 2014-01-02 15:29:08,127 ERROR org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Error in log splitting write thread 2014-01-02 15:29:08,135 ERROR org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Error in log splitting write thread 2014-01-02 15:29:08,371 ERROR org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Couldn't close log at hdfs://localhost:9000/hbase/.META./1028785192/recovered.edits/0000000000000005582.temp 2014-01-02 15:29:08,404 ERROR org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Couldn't close log at hdfs://localhost:9000/hbase/3_webpage/7072d995482044f41db1121a8c2f4b30/recovered.edits/0000000000000005589.temp 2014-01-02 15:29:08,405 ERROR org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Couldn't close log at hdfs://localhost:9000/hbase/3_webpage/ab9bcbedd658fc21eed5794c1d5f2a8c/recovered.edits/0000000000000005591.temp 2014-01-02 15:29:09,461 ERROR org.apache.hadoop.hdfs.DFSClient: Failed to close file /hbase/3_webpage/ab9bcbedd658fc21eed5794c1d5f2a8c/recovered.edits/0000000000000005591.temp 2014-01-02 15:29:09,461 ERROR org.apache.hadoop.hdfs.DFSClient: Failed to close file /hbase/3_webpage/7072d995482044f41db1121a8c2f4b30/recovered.edits/0000000000000005589.temp 2014-01-02 15:29:09,462 ERROR org.apache.hadoop.hdfs.DFSClient: Failed to close file /hbase/.META./1028785192/recovered.edits/0000000000000005582.temp 2014-01-02 15:29:08,082 FATAL org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: WriterThread-2 Got while writing log entry to log 2014-01-02 15:29:08,124 FATAL org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: WriterThread-1 Got while writing log entry to log 2014-01-02 15:29:08,133 FATAL org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: WriterThread-0 Got while writing log entry to log 2014-01-02 15:29:08,409 FATAL org.apache.hadoop.hbase.master.MasterFileSystem: Failed splitting hdfs://localhost:9000/hbase/.logs/localhost,60020,1388670608167 2014-01-02 15:29:08,410 FATAL org.apache.hadoop.hbase.master.HMaster: Shutting down HBase cluster: file system not available 2014-01-02 15:29:03,493 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server serverName=localhost,60020,1388670608167, load=(requests=178, regions=20, usedHeap=594, maxHeap=10667): Replay of HLog required. Forcing server shutdown 2014-01-02 15:28:49,681 WARN org.apache.hadoop.hdfs.DFSClient: Could not get block locations. Source file "/hbase/3_webpage/6f16b76b2f93a85c3c9fb0dc203f44d1/splits/f967df14eca3c7def943cc53fbf9b8a9/ol/5290547552864737575.6f16b76b2f93a85c3c9fb0dc203f44d1" - Aborting... 2014-01-02 15:28:49,680 WARN org.apache.hadoop.hdfs.DFSClient: Could not get block locations. Source file "/hbase/3_webpage/6f16b76b2f93a85c3c9fb0dc203f44d1/splits/f967df14eca3c7def943cc53fbf9b8a9/f/7286765113971143335.6f16b76b2f93a85c3c9fb0dc203f44d1" - Aborting... 2014-01-02 15:28:49,681 WARN org.apache.hadoop.hdfs.DFSClient: Could not get block locations. Source file "/hbase/3_webpage/6f16b76b2f93a85c3c9fb0dc203f44d1/splits/f967df14eca3c7def943cc53fbf9b8a9/mk/8373295057322606514.6f16b76b2f93a85c3c9fb0dc203f44d1" - Aborting... 2014-01-02 15:28:51,815 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 5 on 60020 took 1878 ms appending an edit to hlog; editcount=293, len~=411.9k 2014-01-02 15:28:56,693 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /hbase/3_webpage/d2e85aa82fc5ac7a31fcb384c8a19b28/splits/07f7c06f7dd374934160a226cbd0cd36/f/1050327747216370479.d2e85aa82fc5ac7a31fcb384c8a19b28 could only be replicated to 0 nodes, instead of 1 2014-01-02 15:28:56,693 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for null bad datanode[0] nodes == null 2014-01-02 15:28:56,694 WARN org.apache.hadoop.hdfs.DFSClient: Could not get block locations. Source file "/hbase/3_webpage/d2e85aa82fc5ac7a31fcb384c8a19b28/splits/07f7c06f7dd374934160a226cbd0cd36/f/1050327747216370479.d2e85aa82fc5ac7a31fcb384c8a19b28" - Aborting... 2014-01-02 15:28:56,695 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /hbase/3_webpage/d2e85aa82fc5ac7a31fcb384c8a19b28/splits/07f7c06f7dd374934160a226cbd0cd36/ol/2034350034778366131.d2e85aa82fc5ac7a31fcb384c8a19b28 could only be replicated to 0 nodes, instead of 1 2014-01-02 15:28:56,696 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for null bad datanode[0] nodes == null 2014-01-02 15:28:56,696 WARN org.apache.hadoop.hdfs.DFSClient: Could not get block locations. Source file "/hbase/3_webpage/d2e85aa82fc5ac7a31fcb384c8a19b28/splits/07f7c06f7dd374934160a226cbd0cd36/ol/2034350034778366131.d2e85aa82fc5ac7a31fcb384c8a19b28" - Aborting... 2014-01-02 15:28:56,697 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /hbase/3_webpage/d2e85aa82fc5ac7a31fcb384c8a19b28/splits/07f7c06f7dd374934160a226cbd0cd36/h/2539869235400177968.d2e85aa82fc5ac7a31fcb384c8a19b28 could only be replicated to 0 nodes, instead of 1 2014-01-02 15:28:56,702 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for null bad datanode[0] nodes == null 2014-01-02 15:28:56,702 WARN org.apache.hadoop.hdfs.DFSClient: Could not get block locations. Source file "/hbase/3_webpage/d2e85aa82fc5ac7a31fcb384c8a19b28/splits/07f7c06f7dd374934160a226cbd0cd36/h/2539869235400177968.d2e85aa82fc5ac7a31fcb384c8a19b28" - Aborting... 2014-01-02 15:28:56,702 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /hbase/3_webpage/d2e85aa82fc5ac7a31fcb384c8a19b28/splits/07f7c06f7dd374934160a226cbd0cd36/mtdt/2511732635073603338.d2e85aa82fc5ac7a31fcb384c8a19b28 could only be replicated to 0 nodes, instead of 1 2014-01-02 15:28:56,702 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for null bad datanode[0] nodes == null 2014-01-02 15:28:56,702 WARN org.apache.hadoop.hdfs.DFSClient: Could not get block locations. Source file "/hbase/3_webpage/d2e85aa82fc5ac7a31fcb384c8a19b28/splits/07f7c06f7dd374934160a226cbd0cd36/mtdt/2511732635073603338.d2e85aa82fc5ac7a31fcb384c8a19b28" - Aborting... 2014-01-02 15:28:56,702 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /hbase/3_webpage/d2e85aa82fc5ac7a31fcb384c8a19b28/splits/07f7c06f7dd374934160a226cbd0cd36/mtdt/7051003187947316265.d2e85aa82fc5ac7a31fcb384c8a19b28 could only be replicated to 0 nodes, instead of 1 2014-01-02 15:28:56,703 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for null bad datanode[0] nodes == null 2014-01-02 15:28:56,703 WARN org.apache.hadoop.hdfs.DFSClient: Could not get block locations. Source file "/hbase/3_webpage/d2e85aa82fc5ac7a31fcb384c8a19b28/splits/07f7c06f7dd374934160a226cbd0cd36/mtdt/7051003187947316265.d2e85aa82fc5ac7a31fcb384c8a19b28" - Aborting... 2014-01-02 15:28:56,738 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /hbase/3_webpage/d2e85aa82fc5ac7a31fcb384c8a19b28/splits/07f7c06f7dd374934160a226cbd0cd36/h/7734322749783373345.d2e85aa82fc5ac7a31fcb384c8a19b28 could only be replicated to 0 nodes, instead of 1 2014-01-02 15:28:56,741 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /hbase/3_webpage/d2e85aa82fc5ac7a31fcb384c8a19b28/splits/07f7c06f7dd374934160a226cbd0cd36/mk/5913845763479035889.d2e85aa82fc5ac7a31fcb384c8a19b28 could only be replicated to 0 nodes, instead of 1 2014-01-02 15:28:56,744 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for null bad datanode[0] nodes == null 2014-01-02 15:28:56,744 WARN org.apache.hadoop.hdfs.DFSClient: Could not get block locations. Source file "/hbase/3_webpage/d2e85aa82fc5ac7a31fcb384c8a19b28/splits/07f7c06f7dd374934160a226cbd0cd36/mk/5913845763479035889.d2e85aa82fc5ac7a31fcb384c8a19b28" - Aborting... 2014-01-02 15:28:56,744 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /hbase/3_webpage/d2e85aa82fc5ac7a31fcb384c8a19b28/splits/07f7c06f7dd374934160a226cbd0cd36/mk/3685985664085252295.d2e85aa82fc5ac7a31fcb384c8a19b28 could only be replicated to 0 nodes, instead of 1 2014-01-02 15:28:56,741 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for null bad datanode[0] nodes == null 2014-01-02 15:28:56,745 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for null bad datanode[0] nodes == null 2014-01-02 15:28:56,745 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /hbase/3_webpage/d2e85aa82fc5ac7a31fcb384c8a19b28/splits/07f7c06f7dd374934160a226cbd0cd36/f/2668655577841396270.d2e85aa82fc5ac7a31fcb384c8a19b28 could only be replicated to 0 nodes, instead of 1 2014-01-02 15:28:56,746 WARN org.apache.hadoop.hdfs.DFSClient: Could not get block locations. Source file "/hbase/3_webpage/d2e85aa82fc5ac7a31fcb384c8a19b28/splits/07f7c06f7dd374934160a226cbd0cd36/mk/3685985664085252295.d2e85aa82fc5ac7a31fcb384c8a19b28" - Aborting... 2014-01-02 15:28:56,745 WARN org.apache.hadoop.hdfs.DFSClient: Could not get block locations. Source file "/hbase/3_webpage/d2e85aa82fc5ac7a31fcb384c8a19b28/splits/07f7c06f7dd374934160a226cbd0cd36/h/7734322749783373345.d2e85aa82fc5ac7a31fcb384c8a19b28" - Aborting... 2014-01-02 15:28:56,746 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for null bad datanode[0] nodes == null 2014-01-02 15:28:56,746 WARN org.apache.hadoop.hdfs.DFSClient: Could not get block locations. Source file "/hbase/3_webpage/d2e85aa82fc5ac7a31fcb384c8a19b28/splits/07f7c06f7dd374934160a226cbd0cd36/f/2668655577841396270.d2e85aa82fc5ac7a31fcb384c8a19b28" - Aborting... 2014-01-02 15:29:03,492 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /hbase/3_webpage/c0b264674a5940ac91c851f567244dca/.tmp/3847601395962189673 could only be replicated to 0 nodes, instead of 1 2014-01-02 15:29:03,492 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for null bad datanode[0] nodes == null 2014-01-02 15:29:03,492 WARN org.apache.hadoop.hdfs.DFSClient: Could not get block locations. Source file "/hbase/3_webpage/c0b264674a5940ac91c851f567244dca/.tmp/3847601395962189673" - Aborting... 2014-01-02 15:03:53,894 WARN org.apache.zookeeper.server.NIOServerCnxn: EndOfStreamException: Unable to read additional data from client sessionid 0x1435337da24001d, likely client has closed socket 2014-01-02 15:03:53,894 WARN org.apache.zookeeper.server.NIOServerCnxn: EndOfStreamException: Unable to read additional data from client sessionid 0x1435337da240020, likely client has closed socket 2014-01-02 15:03:53,895 WARN org.apache.zookeeper.server.NIOServerCnxn: EndOfStreamException: Unable to read additional data from client sessionid 0x1435337da24001c, likely client has closed socket 2014-01-02 15:03:53,896 WARN org.apache.zookeeper.server.NIOServerCnxn: EndOfStreamException: Unable to read additional data from client sessionid 0x1435337da24001a, likely client has closed socket 2014-01-02 15:03:53,896 WARN org.apache.zookeeper.server.NIOServerCnxn: EndOfStreamException: Unable to read additional data from client sessionid 0x1435337da24001f, likely client has closed socket 2014-01-02 15:04:35,351 WARN org.apache.zookeeper.server.NIOServerCnxn: EndOfStreamException: Unable to read additional data from client sessionid 0x1435337da24002c, likely client has closed socket 2014-01-02 15:23:00,374 WARN org.apache.zookeeper.server.NIOServerCnxn: EndOfStreamException: Unable to read additional data from client sessionid 0x1435337da24002d, likely client has closed socket === localhost:# hadoop fsck /hbase - openforwrite FSCK started by root from /127.0.0.1 for path /hbase at Thu Jan 02 15:59:11 CET 2014 ......../hbase/.META./1028785192/recovered.edits/0000000000000005582.temp 0 byte s, 0 block(s), OPENFORWRITE: ................................................... ................................./hbase/3_webpage/7072d995482044f41db1121a8c2f4b 30/recovered.edits/0000000000000005589.temp 0 bytes, 0 block(s), OPENFORWRITE: . ..... ............................./hbase/3_webpage/ab9bcbedd658fc21eed5794c1d5f2a8c/r ecovered.edits/0000000000000005591.temp 0 bytes, 0 block(s), OPENFORWRITE: ..... ..../hbase/3_webpage/c0b264674a5940ac91c851f567244dca/.tmp/3847601395962189673 0 bytes, 0 block(s), OPENFORWRITE: .............................................. .............. .................................Status: HEALTHY Total size: 6080372900 B Total dirs: 231 Total files: 233 Total blocks (validated): 272 (avg. block size 22354312 B) Minimally replicated blocks: 272 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 1 Average block replication: 1.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 1 Number of racks: 1 FSCK ended at Thu Jan 02 15:59:11 CET 2014 in 70 milliseconds The filesystem under path '/hbase' is HEALTHY
