HBase gets stuck with "Regions in transition timed out" for .META. and -ROOT-
-----------------------------------------------------------------------------
Key: HBASE-3804
URL: https://issues.apache.org/jira/browse/HBASE-3804
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.2
Environment: CentOs 5.4 x64 kernel 2.6.18-238.5.1.el5
Single node cluster
Reporter: Laurent Chavet
Priority: Blocker
After putting some significant load on the sever (load around of 8 for
approximatively 50 minutes) the following starts showing in the logs and then
hbase is completely stuck: won't recover without a restart (I copy a large
chunk of the master logs before the problem in case it helps; the bottom part
about ROOT and META repeats forever):
9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305484717
2011-04-20 06:19:51,228 INFO
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs --
HDFS-200
2011-04-20 06:19:51,228 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
path=hdfs://hqsnosq
l01:9000/hbase/Property/69e1f895ca9d6824b409015a9f9c03a3/recovered.edits/0000000000000098012
region=69e1f895ca9d6824b409
015a9f9c03a3
2011-04-20 06:19:51,539 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=38 entries from
hdfs://HQSNO
SQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305484717
2011-04-20 06:19:51,540 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog 42 of 43:
hdfs://HQS
NOSQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251,
length=67522024
2011-04-20 06:19:51,540 INFO org.apache.hadoop.hbase.util.FSUtils: Recovering
file hdfs://HQSNOSQL01:9000/hbase/.logs/HQ
SNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251
2011-04-20 06:19:52,543 INFO org.apache.hadoop.hbase.util.FSUtils: Finished
lease recover attempt for hdfs://HQSNOSQL01:
9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251
2011-04-20 06:19:53,336 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=40 entries from
hdfs://HQSNO
SQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251
2011-04-20 06:19:54,894 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog 43 of 43:
hdfs://HQS
NOSQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805,
length=75765855
2011-04-20 06:19:54,906 INFO org.apache.hadoop.hbase.util.FSUtils: Recovering
file hdfs://HQSNOSQL01:9000/hbase/.logs/HQ
SNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805
2011-04-20 06:19:55,108 INFO
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs --
HDFS-200
2011-04-20 06:19:55,108 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
path=hdfs://hqsnosq
l01:9000/hbase/Property/a21bf162e9ff2915ff036f486919c9b5/recovered.edits/0000000000000098062
region=a21bf162e9ff2915ff03
6f486919c9b5
2011-04-20 06:19:55,908 INFO org.apache.hadoop.hbase.util.FSUtils: Finished
lease recover attempt for hdfs://HQSNOSQL01:
9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805
2011-04-20 06:19:56,842 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=43 entries from
hdfs://HQSNO
SQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805
2011-04-20 06:19:56,859 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305143661
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305143661
2011-04-20 06:19:56,863 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305152287
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305152287
2011-04-20 06:19:56,867 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305160979
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305160979
2011-04-20 06:19:56,871 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305170430
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305170430
2011-04-20 06:19:56,875 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305178662
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305178662
2011-04-20 06:19:56,879 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305188551
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305188551
2011-04-20 06:19:56,883 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305196703
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305196703
2011-04-20 06:19:56,887 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305205039
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305205039
2011-04-20 06:19:56,891 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305213412
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305213412
2011-04-20 06:19:56,895 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305220665
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305220665
2011-04-20 06:19:56,899 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305231697
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305231697
2011-04-20 06:19:56,913 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305239174
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305239174
2011-04-20 06:19:56,917 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305248013
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305248013
2011-04-20 06:19:56,921 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305256411
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305256411
2011-04-20 06:19:56,925 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305265663
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305265663
2011-04-20 06:19:56,929 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305274667
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305274667
2011-04-20 06:19:56,933 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305283140
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305283140
2011-04-20 06:19:56,937 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305291307
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305291307
2011-04-20 06:19:56,941 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305300632
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305300632
2011-04-20 06:19:56,945 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305309439
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305309439
2011-04-20 06:19:56,949 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305316405
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305316405
2011-04-20 06:19:56,953 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305322730
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305322730
2011-04-20 06:19:56,957 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305330453
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305330453
2011-04-20 06:19:56,961 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305337020
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305337020
2011-04-20 06:19:56,965 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305345309
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305345309
2011-04-20 06:19:56,969 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305352766
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305352766
2011-04-20 06:19:56,972 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305362335
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305362335
2011-04-20 06:19:56,976 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305370474
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305370474
2011-04-20 06:19:56,980 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305379360
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305379360
2011-04-20 06:19:56,984 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305387434
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305387434
2011-04-20 06:19:56,988 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305396557
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305396557
2011-04-20 06:19:56,992 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305403848
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305403848
2011-04-20 06:19:56,996 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305412950
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305412950
2011-04-20 06:19:57,000 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305420349
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305420349
2011-04-20 06:19:57,004 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305429358
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305429358
2011-04-20 06:19:57,008 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305437383
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305437383
2011-04-20 06:19:57,012 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305446715
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305446715
2011-04-20 06:19:57,016 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305460169
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305460169
2011-04-20 06:19:57,678 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305467377
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305467377
2011-04-20 06:19:57,685 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305476873
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305476873
2011-04-20 06:19:57,689 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305484717
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305484717
2011-04-20 06:19:57,693 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305492251
2011-04-20 06:19:57,697 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805
to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305500805
2011-04-20 06:19:57,701 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Waiting for split writer
threads to
finish
2011-04-20 06:19:57,874 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Split writers finished
2011-04-20 06:19:57,881 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://hqsnosql01:9000/h
base/Property/36d89fc46915d8db56bad45c64a313a3/recovered.edits/0000000000000096979
(wrote 134 edits in 6107ms)
2011-04-20 06:19:57,885 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://hqsnosql01:9000/h
base/Property/4499b405bae1047884256f39b7559376/recovered.edits/0000000000000096571
(wrote 30 edits in 1338ms)
2011-04-20 06:19:57,889 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://hqsnosql01:9000/h
base/Property/465fb6b0a1f2d5306c7bb87145a32be2/recovered.edits/0000000000000096568
(wrote 22 edits in 909ms)
2011-04-20 06:19:57,893 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://hqsnosql01:9000/h
base/Property/4bc3c0bd3a824fb6c7fabbd332ef1009/recovered.edits/0000000000000097151
(wrote 85 edits in 3982ms)
2011-04-20 06:19:57,897 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://hqsnosql01:9000/h
base/Property/4bfff6ceb59cc4b6098eea42b49ee9ed/recovered.edits/0000000000000097336
(wrote 117 edits in 4810ms)
2011-04-20 06:19:57,901 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://hqsnosql01:9000/h
base/Property/52dee69cbe0890b54aee56bda54577c9/recovered.edits/0000000000000096567
(wrote 63 edits in 4445ms)
2011-04-20 06:19:57,905 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://hqsnosql01:9000/h
base/Property/539e4afe8064897395d4dc891605e702/recovered.edits/0000000000000097127
(wrote 53 edits in 3433ms)
2011-04-20 06:19:57,909 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://hqsnosql01:9000/h
base/Property/5728cb0bb67c7c3d8701fc0679db97b9/recovered.edits/0000000000000097473
(wrote 134 edits in 6623ms)
2011-04-20 06:19:57,913 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://hqsnosql01:9000/h
base/Property/5f92d0c1defc108140df27322257584d/recovered.edits/0000000000000096672
(wrote 77 edits in 7211ms)
2011-04-20 06:19:57,917 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://hqsnosql01:9000/h
base/Property/69e1f895ca9d6824b409015a9f9c03a3/recovered.edits/0000000000000098012
(wrote 10 edits in 306ms)
2011-04-20 06:19:57,921 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://hqsnosql01:9000/h
base/Property/70746a106ab152f290264d0b2b0f773c/recovered.edits/0000000000000096722
(wrote 117 edits in 7245ms)
2011-04-20 06:19:57,925 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://hqsnosql01:9000/h
base/Property/906e3283419ae67052c34e31cc40ff05/recovered.edits/0000000000000097951
(wrote 31 edits in 2389ms)
2011-04-20 06:19:57,929 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://hqsnosql01:9000/h
base/Property/94112f821b90271f9d693d5505d5eebf/recovered.edits/0000000000000097692
(wrote 71 edits in 4771ms)
2011-04-20 06:19:57,933 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://hqsnosql01:9000/h
base/Property/9f519dfd7370990d017e7f6d187968d3/recovered.edits/0000000000000096697
(wrote 77 edits in 5551ms)
2011-04-20 06:19:57,937 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://hqsnosql01:9000/h
base/Property/9f60e52d384d0fd9dfc2a2f3be42ac43/recovered.edits/0000000000000097877
(wrote 24 edits in 840ms)
2011-04-20 06:19:58,054 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://hqsnosql01:9000/h
base/Property/a21bf162e9ff2915ff036f486919c9b5/recovered.edits/0000000000000098062
(wrote 8 edits in 1860ms)
2011-04-20 06:19:58,061 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://hqsnosql01:9000/h
base/Property/abc26969d71b22de80c7e889cc9a6c12/recovered.edits/0000000000000096576
(wrote 5 edits in 281ms)
2011-04-20 06:19:58,065 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://hqsnosql01:9000/h
base/Property/b5d7c2bd8383ea8871890f40b9827dcf/recovered.edits/0000000000000097554
(wrote 125 edits in 5240ms)
2011-04-20 06:19:58,073 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://hqsnosql01:9000/h
base/Property/bf06b369cf84e5bccdd4e845636a2af6/recovered.edits/0000000000000096839
(wrote 1 edits in 17ms)
2011-04-20 06:19:58,077 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://hqsnosql01:9000/h
base/Property/cf46d330cdaa1120c8e1a5fcf1899b0f/recovered.edits/0000000000000097442
(wrote 1 edits in 850ms)
2011-04-20 06:19:58,081 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://hqsnosql01:9000/h
base/Property/d850f84bfdfa28c63934a0be5a41257e/recovered.edits/0000000000000096572
(wrote 53 edits in 2467ms)
2011-04-20 06:19:58,085 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://hqsnosql01:9000/h
base/Property/d9b94307130a7096032b44b3aa18d3ea/recovered.edits/0000000000000096795
(wrote 155 edits in 10534ms)
2011-04-20 06:19:58,088 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://hqsnosql01:9000/h
base/Property/da6540fdaae565a6d04cd78a145240dd/recovered.edits/0000000000000097580
(wrote 53 edits in 2992ms)
2011-04-20 06:19:58,092 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://hqsnosql01:9000/h
base/Property/df02c5d923c466c5b89337fecee920cf/recovered.edits/0000000000000096924
(wrote 1 edits in 25ms)
2011-04-20 06:19:58,096 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://hqsnosql01:9000/h
base/Property/e90c1c020e3148756a57d977133e31fc/recovered.edits/0000000000000096569
(wrote 96 edits in 3862ms)
2011-04-20 06:19:58,097 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: hlog file splitting
completed in 866
02 ms for hdfs://hqsnosql01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146
2011-04-20 06:19:58,102 INFO
org.apache.hadoop.hbase.catalog.RootLocationEditor: Unsetting ROOT region
location in ZooKe
eper
2011-04-20 06:19:58,113 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:60000-0x12f72de7aee0000 Creating (or up
dating) unassigned node for 70236052 with OFFLINE state
2011-04-20 06:19:58,133 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:60000-0x12f72de7aee0000 Creating (or up
dating) unassigned node for 1028785192 with OFFLINE state
2011-04-20 06:19:58,139 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Handling transition=M_ZK_REGION_OFFLINE,
server=HQSNOSQL01:60000, region=1028785192/.META.
2011-04-20 06:20:36,363 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Regions in transition timed out: .META.,
,1.1028785192 state=OFFLINE, ts=1303305598133
2011-04-20 06:20:36,364 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Region has been OFFLINE for too long, rea
ssigning .META.,,1.1028785192 to a random server
2011-04-20 06:20:36,364 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Regions in transition timed out: -ROOT-,
,0.70236052 state=OFFLINE, ts=1303305598113
2011-04-20 06:20:36,364 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Region has been OFFLINE for too long, rea
ssigning -ROOT-,,0.70236052 to a random server
2011-04-20 06:20:36,364 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Forcing OFFLINE; was=.META.,,1.102878519
2 state=OFFLINE, ts=1303305598133
2011-04-20 06:20:36,364 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Forcing OFFLINE; was=-ROOT-,,0.70236052
state=OFFLINE, ts=1303305598113
2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Regions in transition timed out: .META.,
,1.1028785192 state=OFFLINE, ts=1303305636364
2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Region has been OFFLINE for too long, rea
ssigning .META.,,1.1028785192 to a random server
2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Regions in transition timed out: -ROOT-,
,0.70236052 state=OFFLINE, ts=1303305636364
2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Region has been OFFLINE for too long, rea
ssigning -ROOT-,,0.70236052 to a random server
2011-04-20 06:21:06,374 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Forcing OFFLINE; was=.META.,,1.102878519
2 state=OFFLINE, ts=1303305636364
2011-04-20 06:21:06,374 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Forcing OFFLINE; was=-ROOT-,,0.70236052
state=OFFLINE, ts=1303305636364
2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Regions in transition timed out: .META.,
,1.1028785192 state=OFFLINE, ts=1303305666374
2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Region has been OFFLINE for too long, rea
ssigning .META.,,1.1028785192 to a random server
2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Regions in transition timed out: -ROOT-,
,0.70236052 state=OFFLINE, ts=1303305666374
2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Region has been OFFLINE for too long, rea
ssigning -ROOT-,,0.70236052 to a random server
2011-04-20 06:21:36,385 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Forcing OFFLINE; was=.META.,,1.102878519
2 state=OFFLINE, ts=1303305666374
2011-04-20 06:21:36,385 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Forcing OFFLINE; was=-ROOT-,,0.70236052
state=OFFLINE, ts=1303305666374
2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Regions in transition timed out: .META.,
,1.1028785192 state=OFFLINE, ts=1303305696385
2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Region has been OFFLINE for too long, rea
ssigning .META.,,1.1028785192 to a random server
2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Regions in transition timed out: -ROOT-,
,0.70236052 state=OFFLINE, ts=1303305696385
2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Region has been OFFLINE for too long, rea
ssigning -ROOT-,,0.70236052 to a random server
2011-04-20 06:22:06,393 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Forcing OFFLINE; was=.META.,,1.102878519
2 state=OFFLINE, ts=1303305696385
2011-04-20 06:22:06,393 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Forcing OFFLINE; was=-ROOT-,,0.70236052
state=OFFLINE, ts=1303305696385
2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Regions in transition timed out: .META.,
,1.1028785192 state=OFFLINE, ts=1303305726393
2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Region has been OFFLINE for too long, rea
ssigning .META.,,1.1028785192 to a random server
2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Regions in transition timed out: -ROOT-,
,0.70236052 state=OFFLINE, ts=1303305726394
2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Region has been OFFLINE for too long, rea
ssigning -ROOT-,,0.70236052 to a random server
2011-04-20 06:22:36,402 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Forcing OFFLINE; was=.META.,,1.102878519
2 state=OFFLINE, ts=1303305726393
2011-04-20 06:22:36,402 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Forcing OFFLINE; was=-ROOT-,,0.70236052
state=OFFLINE, ts=1303305726394
2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Regions in transition timed out: .META.,
,1.1028785192 state=OFFLINE, ts=1303305756402
2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Region has been OFFLINE for too long, rea
ssigning .META.,,1.1028785192 to a random server
2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Regions in transition timed out: -ROOT-,
,0.70236052 state=OFFLINE, ts=1303305756402
2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Region has been OFFLINE for too long, rea
ssigning -ROOT-,,0.70236052 to a random server
2011-04-20 06:23:06,412 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Forcing OFFLINE; was=.META.,,1.102878519
2 state=OFFLINE, ts=1303305756402
2011-04-20 06:23:06,412 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Forcing OFFLINE; was=-ROOT-,,0.70236052
state=OFFLINE, ts=1303305756402
2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Regions in transition timed out: .META.,
,1.1028785192 state=OFFLINE, ts=1303305786412
2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Region has been OFFLINE for too long, rea
ssigning .META.,,1.1028785192 to a random server
2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Regions in transition timed out: -ROOT-,
,0.70236052 state=OFFLINE, ts=1303305786412
2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Region has been OFFLINE for too long, rea
ssigning -ROOT-,,0.70236052 to a random server
2011-04-20 06:23:36,420 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Forcing OFFLINE; was=.META.,,1.102878519
2 state=OFFLINE, ts=1303305786412
2011-04-20 06:23:36,420 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Forcing OFFLINE; was=-ROOT-,,0.70236052
state=OFFLINE, ts=1303305786412
2011-04-20 06:24:06,429 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Regions in transition timed out: .META.,
,1.1028785192 state=OFFLINE, ts=1303305816420
2011-04-20 06:24:06,430 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Region has been OFFLINE for too long, rea
ssigning .META.,,1.1028785192 to a random server
2011-04-20 06:24:06,430 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Regions in transition timed out: -ROOT-,
,0.70236052 state=OFFLINE, ts=1303305816420
2011-04-20 06:24:06,430 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Region has been OFFLINE for too long, rea
ssigning -ROOT-,,0.70236052 to a random server
2011-04-20 06:24:06,430 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Forcing OFFLINE; was=.META.,,1.102878519
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
