That looks sick. Different log files can't close? Enable DEBUG in your logs. See FAQ for how. You sure your HDFS healthy? Is it even working? St.Ack
On Tue, May 26, 2009 at 10:06 AM, llpind <[email protected]> wrote: > > All, > > I tailed one of the region servers that seems to be handling most of the > load. Here is what I saw: > > 2009-05-26 10:00:21,015 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:21,419 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:21,823 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:22,227 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:22,631 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:23,035 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:23,439 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:23,708 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer > Exception: java.net.SocketTimeoutException: 5000 millis timeout while > waiting for channel to be ready for write. ch : > java.nio.channels.SocketChannel[connected local=/192.168.240.175:53828 > remote=/192.168.240.179:50010] > at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:162) > at > org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146) > at > org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107) > at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105) > at java.io.DataOutputStream.write(DataOutputStream.java:90) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2209) > > 2009-05-26 10:00:23,708 WARN org.apache.hadoop.hdfs.DFSClient: Error > Recovery for block blk_1087586329384370907_1077 bad datanode[0] > 192.168.240.179:50010 > 2009-05-26 10:00:23,843 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:24,247 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:24,651 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:25,055 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:25,459 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:25,863 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:26,267 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:26,671 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:27,075 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:27,479 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:27,883 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:28,287 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:28,692 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:29,096 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:29,500 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:29,904 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:30,308 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:30,712 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:31,116 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:31,520 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:31,924 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:32,328 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:32,732 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:33,136 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:33,540 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:33,944 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:34,332 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:34,348 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:34,736 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:34,752 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:35,140 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:35,156 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:35,544 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:35,560 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:35,948 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:35,964 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:36,352 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:36,368 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:36,756 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:36,772 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:37,160 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:37,176 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:37,564 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:37,580 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:37,968 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:37,984 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:38,372 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:38,388 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:38,776 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:38,792 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:39,180 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:39,196 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:39,584 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:39,600 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:39,988 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:40,004 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:40,392 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:40,408 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:40,796 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:40,812 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:41,200 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:41,216 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:41,604 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:41,620 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:42,008 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:42,024 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:42,412 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:42,464 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:42,816 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:42,868 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:43,220 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:43,272 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:43,624 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:43,676 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:44,028 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:44,080 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:44,432 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:44,485 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:44,837 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:44,888 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:45,241 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:45,293 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:45,645 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:45,697 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:46,049 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:46,101 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:46,453 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:46,505 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:46,857 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:46,909 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:47,261 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:47,501 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:47,665 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:47,905 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:48,069 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:48,309 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:48,473 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:48,713 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:48,877 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:49,117 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:49,281 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:49,521 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:49,685 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:49,925 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:50,089 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:50,329 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:50,493 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:50,733 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:50,897 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:51,137 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:51,301 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:51,541 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:51,705 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:51,945 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:52,109 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:52,349 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:52,513 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:52,753 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:52,917 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:53,157 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:53,321 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:53,561 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:53,725 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:53,965 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:54,133 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:54,369 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:54,537 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:54,773 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:54,941 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:55,177 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:55,345 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:55,581 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:55,749 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:55,985 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:56,153 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:56,389 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:56,557 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:56,793 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:56,961 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:57,317 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:57,365 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:57,721 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:57,769 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:58,125 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:58,173 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:58,529 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:58,577 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:58,933 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:58,981 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:59,337 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:59,385 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:00:59,741 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:00:59,789 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:01:00,145 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:01:00,193 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:01:00,549 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:01:00,597 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:01:00,954 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:01:01,002 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:01:01,358 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:01:01,406 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:01:01,762 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:01:01,810 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:01:02,166 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:01:02,214 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:01:02,570 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:01:02,618 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:01:02,974 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:01:03,022 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:01:03,378 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:01:03,426 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:01:03,782 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 > retrying... > 2009-05-26 10:01:03,830 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/4259450164056335713/index > retrying... > 2009-05-26 10:01:03,849 ERROR > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction/Split > failed for region tableA,ROW_KEY,1243357190459 > java.io.IOException: All datanodes 192.168.240.179:50010 are bad. > Aborting... > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2444) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:1996) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2160) > 2009-05-26 10:01:03,891 INFO org.apache.hadoop.hbase.regionserver.HLog: > Closed > hdfs://ats181:54310/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357129598, > entries=100002. New log writer: > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357263846 > 2009-05-26 10:01:03,891 INFO org.apache.hadoop.hbase.regionserver.HLog: > removing old log file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357078715 whose > highest sequence/edit id is 1175008 > 2009-05-26 10:01:37,493 INFO org.apache.hadoop.hbase.regionserver.HLog: > Closed > hdfs://ats181:54310/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473, > entries=105000. New log writer: > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357297485 > 2009-05-26 10:01:39,353 INFO org.apache.hadoop.hbase.regionserver.HRegion: > starting compaction on region tableA,ROW_KEY,1243357190459 > 2009-05-26 10:01:40,903 INFO org.apache.hadoop.hbase.regionserver.HLog: > Closed > hdfs://ats181:54310/hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357263846, > entries=100001. New log writer: > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357300890 > 2009-05-26 10:01:40,904 INFO org.apache.hadoop.hbase.regionserver.HLog: > removing old log file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357129598 whose > highest sequence/edit id is 1275010 > 2009-05-26 10:01:40,912 INFO org.apache.hadoop.hbase.regionserver.HLog: > removing old log file > /hbase/log_192.168.240.175_1243356336827_60020/hlog.dat.1243357169473 whose > highest sequence/edit id is 1380010 > 2009-05-26 10:01:59,165 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/6229032204546731894/data > retrying... > 2009-05-26 10:01:59,569 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/6229032204546731894/data > retrying... > 2009-05-26 10:01:59,973 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/6229032204546731894/data > retrying... > 2009-05-26 10:02:00,377 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/6229032204546731894/data > retrying... > 2009-05-26 10:02:00,781 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/6229032204546731894/data > retrying... > 2009-05-26 10:02:01,185 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/6229032204546731894/data > retrying... > 2009-05-26 10:02:01,589 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/6229032204546731894/data > retrying... > 2009-05-26 10:02:01,993 INFO org.apache.hadoop.hdfs.DFSClient: Could not > complete file > /hbase/tableA/compaction.dir/336579621/link/mapfiles/6229032204546731894/data > retrying... > > -- > View this message in context: > http://www.nabble.com/HBase-looses-regions.-tp23657983p23727376.html > Sent from the HBase User mailing list archive at Nabble.com. > >
