This is similar to just-previous note. We are trying to roll the log
but hdfs is failing us ("bad datanode').
St.Ack
On Tue, May 26, 2009 at 10:39 AM, llpind <[email protected]> wrote:
>
> Finally failed between 7M-8M records. below is the last tail output. The
> other two region server don't have much activity in the logs, but i can post
> those if necessary.
>
> ===================================================
>
> 2009-05-26 10:28:06,550 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_3233282543359573226_1303 bad datanode[0]
> 192.168.240.175:50010
> 2009-05-26 10:28:06,550 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_3233282543359573226_1303 in pipeline
> 192.168.240.175:50010, 192.168.240.180:50010: bad datanode
> 192.168.240.175:50010
> 2009-05-26 10:28:11,714 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
> Exception: java.net.SocketTimeoutException: 5000 millis timeout while
> waiting for channel to be ready for write. ch :
> java.nio.channels.SocketChannel[connected local=/192.168.240.175:60733
> remote=/192.168.240.180:50010]
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:162)
> at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2209)
>
> 2009-05-26 10:28:11,715 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_3233282543359573226_1311 bad datanode[0]
> 192.168.240.180:50010
> 2009-05-26 10:28:11,715 FATAL
> org.apache.hadoop.hbase.regionserver.LogRoller: Log rolling failed with ioe:
> java.io.IOException: All datanodes 192.168.240.180:50010 are bad.
> Aborting...
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2444)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:1996)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2160)
> 2009-05-26 10:28:11,716 FATAL org.apache.hadoop.hbase.regionserver.HLog:
> Could not append. Requesting close of log
> java.io.IOException: All datanodes 192.168.240.180:50010 are bad.
> Aborting...
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2444)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:1996)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2160)
> 2009-05-26 10:28:11,717 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: java.io.IOException: All
> datanodes 192.168.240.180:50010 are bad. Aborting...
> 2009-05-26 10:28:11,726 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
> request=0.0, regions=2, stores=4, storefiles=6, storefileIndexSize=0,
> memcacheSize=40, usedHeap=94, maxHeap=2999
> 2009-05-26 10:28:11,726 INFO org.apache.hadoop.hbase.regionserver.LogRoller:
> LogRoller exiting.
> 2009-05-26 10:28:11,726 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 5 on 60020, call batchUpdates([...@41fb8e4,
> [Lorg.apache.hadoop.hbase.io.BatchUpdate;@3ea382d9) from
> 192.168.240.152:17086: error: java.io.IOException: All datanodes
> 192.168.240.180:50010 are bad. Aborting...
> java.io.IOException: All datanodes 192.168.240.180:50010 are bad.
> Aborting...
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2444)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:1996)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2160)
> 2009-05-26 10:28:12,894 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> server on 60020
> 2009-05-26 10:28:12,895 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 2 on 60020: exiting
> 2009-05-26 10:28:12,895 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 6 on 60020: exiting
> 2009-05-26 10:28:12,895 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC
> Server listener on 60020
> 2009-05-26 10:28:12,896 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 0 on 60020: exiting
> 2009-05-26 10:28:12,896 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 1 on 60020: exiting
> 2009-05-26 10:28:12,896 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 3 on 60020: exiting
> 2009-05-26 10:28:12,897 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 5 on 60020: exiting
> 2009-05-26 10:28:12,897 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 4 on 60020: exiting
> 2009-05-26 10:28:12,898 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 7 on 60020: exiting
> 2009-05-26 10:28:12,898 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC
> Server Responder
> 2009-05-26 10:28:12,898 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 8 on 60020: exiting
> 2009-05-26 10:28:12,898 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 9 on 60020: exiting
> 2009-05-26 10:28:12,898 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
> 2009-05-26 10:28:12,901 INFO org.mortbay.util.ThreadedServer: Stopping
> Acceptor ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=60030]
> 2009-05-26 10:28:12,908 INFO org.mortbay.http.SocketListener: Stopped
> SocketListener on 0.0.0.0:60030
> 2009-05-26 10:28:13,345 INFO org.mortbay.util.Container: Stopped
> HttpContext[/logs,/logs]
> 2009-05-26 10:28:13,346 INFO org.mortbay.util.Container: Stopped
> org.mortbay.jetty.servlet.webapplicationhand...@6ad3c65d
> 2009-05-26 10:28:13,687 INFO org.mortbay.util.Container: Stopped
> WebApplicationContext[/static,/static]
> 2009-05-26 10:28:13,687 INFO org.mortbay.util.Container: Stopped
> org.mortbay.jetty.servlet.webapplicationhand...@3adec8b3
> 2009-05-26 10:28:14,039 INFO org.mortbay.util.Container: Stopped
> WebApplicationContext[/,/]
> 2009-05-26 10:28:14,040 INFO org.mortbay.util.Container: Stopped
> org.mortbay.jetty.ser...@6e79839
> 2009-05-26 10:28:14,040 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread exiting
> 2009-05-26 10:28:14,040 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: On abort, closed hlog
> 2009-05-26 10:28:14,040 INFO
> org.apache.hadoop.hbase.regionserver.MemcacheFlusher:
> regionserver/0.0.0.0:60020.cacheFlusher exiting
> 2009-05-26 10:28:14,040 INFO
> org.apache.hadoop.hbase.regionserver.LogFlusher:
> regionserver/0.0.0.0:60020.logFlusher exiting
> 2009-05-26 10:28:14,040 INFO
> org.apache.hadoop.hbase.regionserver.CompactSplitThread:
> regionserver/0.0.0.0:60020.compactor exiting
> 2009-05-26 10:28:14,040 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker:
> regionserver/0.0.0.0:60020.majorCompactionChecker exiting
> 2009-05-26 10:28:14,041 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Closed TableA,ROW_KEY,1243357190459
> 2009-05-26 10:28:14,041 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Closed TableA,,1243357190459
> 2009-05-26 10:28:14,041 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server at:
> 192.168.240.175:60020
> 2009-05-26 10:28:14,044 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> regionserver/0.0.0.0:60020 exiting
> 2009-05-26 10:28:14,270 INFO org.apache.hadoop.hbase.Leases:
> regionserver/0.0.0.0:60020.leaseChecker closing leases
> 2009-05-26 10:28:14,271 INFO org.apache.hadoop.hbase.Leases:
> regionserver/0.0.0.0:60020.leaseChecker closed leases
> 2009-05-26 10:28:14,273 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Starting shutdown
> thread.
> 2009-05-26 10:28:14,273 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread complete
>
>
> ===================================================
> --
> View this message in context:
> http://www.nabble.com/HBase-looses-regions.-tp23657983p23727987.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>
