[
https://issues.apache.org/jira/browse/HBASE-3804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13022831#comment-13022831
]
Jean-Daniel Cryans commented on HBASE-3804:
-------------------------------------------
So how it works: for some reason your region server died. A region server is
what carries and serves the regions (so the data). One thing that's not clear
to me is if you are running in "standalone" mode or "pseudo-distributed" mode.
If the former, then yeah I agree that it's a bit dumb that it locks up like
that because you cannot restart the region server _inside_ the master process.
If you were using the latter, then well your region server died and you need to
restart it.
What the master does when a node dies is that it splits the write ahead logs
(that's what's going on in the beginning of your log) and reassigns the
regions. In your case, there's no one to reassign the regions to. The master
cannot carry the regions, so it's waiting for a new region server to come in.
On to why the region server died, there should be a hint in the log. Something
on the FATAL log level. Probably GC pauses.
BTW the book has this to say on single node clusters: "Do not use this
configuration for production nor for evaluating HBase performance."
> HBase gets stuck with "Regions in transition timed out" for .META. and -ROOT-
> -----------------------------------------------------------------------------
>
> Key: HBASE-3804
> URL: https://issues.apache.org/jira/browse/HBASE-3804
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.2
> Environment: CentOs 5.4 x64 kernel 2.6.18-238.5.1.el5
> Single node cluster
> Reporter: Laurent Chavet
> Priority: Blocker
>
> After putting some significant load on the sever (load around of 8 for
> approximatively 50 minutes) the following starts showing in the logs and then
> hbase is completely stuck: won't recover without a restart (I copy a large
> chunk of the master logs before the problem in case it helps; the bottom part
> about ROOT and META repeats forever):
> 9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305484717
> 2011-04-20 06:19:51,228 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs
> -- HDFS-200
> 2011-04-20 06:19:51,228 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://hqsnosq
> l01:9000/hbase/Property/69e1f895ca9d6824b409015a9f9c03a3/recovered.edits/0000000000000098012
> region=69e1f895ca9d6824b409
> 015a9f9c03a3
> 2011-04-20 06:19:51,539 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=38 entries from
> hdfs://HQSNO
> SQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305484717
> 2011-04-20 06:19:51,540 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog 42 of
> 43: hdfs://HQS
> NOSQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251,
> length=67522024
> 2011-04-20 06:19:51,540 INFO org.apache.hadoop.hbase.util.FSUtils: Recovering
> file hdfs://HQSNOSQL01:9000/hbase/.logs/HQ
> SNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251
> 2011-04-20 06:19:52,543 INFO org.apache.hadoop.hbase.util.FSUtils: Finished
> lease recover attempt for hdfs://HQSNOSQL01:
> 9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251
> 2011-04-20 06:19:53,336 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=40 entries from
> hdfs://HQSNO
> SQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251
> 2011-04-20 06:19:54,894 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog 43 of
> 43: hdfs://HQS
> NOSQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805,
> length=75765855
> 2011-04-20 06:19:54,906 INFO org.apache.hadoop.hbase.util.FSUtils: Recovering
> file hdfs://HQSNOSQL01:9000/hbase/.logs/HQ
> SNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805
> 2011-04-20 06:19:55,108 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs
> -- HDFS-200
> 2011-04-20 06:19:55,108 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://hqsnosq
> l01:9000/hbase/Property/a21bf162e9ff2915ff036f486919c9b5/recovered.edits/0000000000000098062
> region=a21bf162e9ff2915ff03
> 6f486919c9b5
> 2011-04-20 06:19:55,908 INFO org.apache.hadoop.hbase.util.FSUtils: Finished
> lease recover attempt for hdfs://HQSNOSQL01:
> 9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805
> 2011-04-20 06:19:56,842 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=43 entries from
> hdfs://HQSNO
> SQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805
> 2011-04-20 06:19:56,859 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305143661
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305143661
> 2011-04-20 06:19:56,863 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305152287
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305152287
> 2011-04-20 06:19:56,867 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305160979
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305160979
> 2011-04-20 06:19:56,871 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305170430
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305170430
> 2011-04-20 06:19:56,875 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305178662
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305178662
> 2011-04-20 06:19:56,879 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305188551
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305188551
> 2011-04-20 06:19:56,883 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305196703
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305196703
> 2011-04-20 06:19:56,887 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305205039
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305205039
> 2011-04-20 06:19:56,891 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305213412
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305213412
> 2011-04-20 06:19:56,895 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305220665
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305220665
> 2011-04-20 06:19:56,899 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305231697
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305231697
> 2011-04-20 06:19:56,913 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305239174
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305239174
> 2011-04-20 06:19:56,917 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305248013
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305248013
> 2011-04-20 06:19:56,921 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305256411
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305256411
> 2011-04-20 06:19:56,925 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305265663
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305265663
> 2011-04-20 06:19:56,929 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305274667
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305274667
> 2011-04-20 06:19:56,933 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305283140
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305283140
> 2011-04-20 06:19:56,937 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305291307
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305291307
> 2011-04-20 06:19:56,941 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305300632
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305300632
> 2011-04-20 06:19:56,945 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305309439
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305309439
> 2011-04-20 06:19:56,949 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305316405
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305316405
> 2011-04-20 06:19:56,953 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305322730
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305322730
> 2011-04-20 06:19:56,957 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305330453
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305330453
> 2011-04-20 06:19:56,961 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305337020
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305337020
> 2011-04-20 06:19:56,965 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305345309
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305345309
> 2011-04-20 06:19:56,969 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305352766
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305352766
> 2011-04-20 06:19:56,972 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305362335
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305362335
> 2011-04-20 06:19:56,976 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305370474
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305370474
> 2011-04-20 06:19:56,980 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305379360
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305379360
> 2011-04-20 06:19:56,984 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305387434
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305387434
> 2011-04-20 06:19:56,988 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305396557
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305396557
> 2011-04-20 06:19:56,992 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305403848
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305403848
> 2011-04-20 06:19:56,996 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305412950
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305412950
> 2011-04-20 06:19:57,000 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305420349
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305420349
> 2011-04-20 06:19:57,004 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305429358
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305429358
> 2011-04-20 06:19:57,008 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305437383
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305437383
> 2011-04-20 06:19:57,012 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305446715
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305446715
> 2011-04-20 06:19:57,016 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305460169
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305460169
> 2011-04-20 06:19:57,678 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305467377
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305467377
> 2011-04-20 06:19:57,685 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305476873
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305476873
> 2011-04-20 06:19:57,689 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305484717
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305484717
> 2011-04-20 06:19:57,693 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305492251
> 2011-04-20 06:19:57,697 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log
> hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805
> to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305500805
> 2011-04-20 06:19:57,701 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Waiting for split
> writer threads to
> finish
> 2011-04-20 06:19:57,874 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Split writers finished
> 2011-04-20 06:19:57,881 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://hqsnosql01:9000/h
> base/Property/36d89fc46915d8db56bad45c64a313a3/recovered.edits/0000000000000096979
> (wrote 134 edits in 6107ms)
> 2011-04-20 06:19:57,885 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://hqsnosql01:9000/h
> base/Property/4499b405bae1047884256f39b7559376/recovered.edits/0000000000000096571
> (wrote 30 edits in 1338ms)
> 2011-04-20 06:19:57,889 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://hqsnosql01:9000/h
> base/Property/465fb6b0a1f2d5306c7bb87145a32be2/recovered.edits/0000000000000096568
> (wrote 22 edits in 909ms)
> 2011-04-20 06:19:57,893 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://hqsnosql01:9000/h
> base/Property/4bc3c0bd3a824fb6c7fabbd332ef1009/recovered.edits/0000000000000097151
> (wrote 85 edits in 3982ms)
> 2011-04-20 06:19:57,897 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://hqsnosql01:9000/h
> base/Property/4bfff6ceb59cc4b6098eea42b49ee9ed/recovered.edits/0000000000000097336
> (wrote 117 edits in 4810ms)
> 2011-04-20 06:19:57,901 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://hqsnosql01:9000/h
> base/Property/52dee69cbe0890b54aee56bda54577c9/recovered.edits/0000000000000096567
> (wrote 63 edits in 4445ms)
> 2011-04-20 06:19:57,905 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://hqsnosql01:9000/h
> base/Property/539e4afe8064897395d4dc891605e702/recovered.edits/0000000000000097127
> (wrote 53 edits in 3433ms)
> 2011-04-20 06:19:57,909 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://hqsnosql01:9000/h
> base/Property/5728cb0bb67c7c3d8701fc0679db97b9/recovered.edits/0000000000000097473
> (wrote 134 edits in 6623ms)
> 2011-04-20 06:19:57,913 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://hqsnosql01:9000/h
> base/Property/5f92d0c1defc108140df27322257584d/recovered.edits/0000000000000096672
> (wrote 77 edits in 7211ms)
> 2011-04-20 06:19:57,917 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://hqsnosql01:9000/h
> base/Property/69e1f895ca9d6824b409015a9f9c03a3/recovered.edits/0000000000000098012
> (wrote 10 edits in 306ms)
> 2011-04-20 06:19:57,921 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://hqsnosql01:9000/h
> base/Property/70746a106ab152f290264d0b2b0f773c/recovered.edits/0000000000000096722
> (wrote 117 edits in 7245ms)
> 2011-04-20 06:19:57,925 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://hqsnosql01:9000/h
> base/Property/906e3283419ae67052c34e31cc40ff05/recovered.edits/0000000000000097951
> (wrote 31 edits in 2389ms)
> 2011-04-20 06:19:57,929 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://hqsnosql01:9000/h
> base/Property/94112f821b90271f9d693d5505d5eebf/recovered.edits/0000000000000097692
> (wrote 71 edits in 4771ms)
> 2011-04-20 06:19:57,933 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://hqsnosql01:9000/h
> base/Property/9f519dfd7370990d017e7f6d187968d3/recovered.edits/0000000000000096697
> (wrote 77 edits in 5551ms)
> 2011-04-20 06:19:57,937 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://hqsnosql01:9000/h
> base/Property/9f60e52d384d0fd9dfc2a2f3be42ac43/recovered.edits/0000000000000097877
> (wrote 24 edits in 840ms)
> 2011-04-20 06:19:58,054 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://hqsnosql01:9000/h
> base/Property/a21bf162e9ff2915ff036f486919c9b5/recovered.edits/0000000000000098062
> (wrote 8 edits in 1860ms)
> 2011-04-20 06:19:58,061 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://hqsnosql01:9000/h
> base/Property/abc26969d71b22de80c7e889cc9a6c12/recovered.edits/0000000000000096576
> (wrote 5 edits in 281ms)
> 2011-04-20 06:19:58,065 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://hqsnosql01:9000/h
> base/Property/b5d7c2bd8383ea8871890f40b9827dcf/recovered.edits/0000000000000097554
> (wrote 125 edits in 5240ms)
> 2011-04-20 06:19:58,073 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://hqsnosql01:9000/h
> base/Property/bf06b369cf84e5bccdd4e845636a2af6/recovered.edits/0000000000000096839
> (wrote 1 edits in 17ms)
> 2011-04-20 06:19:58,077 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://hqsnosql01:9000/h
> base/Property/cf46d330cdaa1120c8e1a5fcf1899b0f/recovered.edits/0000000000000097442
> (wrote 1 edits in 850ms)
> 2011-04-20 06:19:58,081 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://hqsnosql01:9000/h
> base/Property/d850f84bfdfa28c63934a0be5a41257e/recovered.edits/0000000000000096572
> (wrote 53 edits in 2467ms)
> 2011-04-20 06:19:58,085 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://hqsnosql01:9000/h
> base/Property/d9b94307130a7096032b44b3aa18d3ea/recovered.edits/0000000000000096795
> (wrote 155 edits in 10534ms)
> 2011-04-20 06:19:58,088 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://hqsnosql01:9000/h
> base/Property/da6540fdaae565a6d04cd78a145240dd/recovered.edits/0000000000000097580
> (wrote 53 edits in 2992ms)
> 2011-04-20 06:19:58,092 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://hqsnosql01:9000/h
> base/Property/df02c5d923c466c5b89337fecee920cf/recovered.edits/0000000000000096924
> (wrote 1 edits in 25ms)
> 2011-04-20 06:19:58,096 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://hqsnosql01:9000/h
> base/Property/e90c1c020e3148756a57d977133e31fc/recovered.edits/0000000000000096569
> (wrote 96 edits in 3862ms)
> 2011-04-20 06:19:58,097 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: hlog file splitting
> completed in 866
> 02 ms for hdfs://hqsnosql01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146
> 2011-04-20 06:19:58,102 INFO
> org.apache.hadoop.hbase.catalog.RootLocationEditor: Unsetting ROOT region
> location in ZooKe
> eper
> 2011-04-20 06:19:58,113 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:60000-0x12f72de7aee0000 Creating (or up
> dating) unassigned node for 70236052 with OFFLINE state
> 2011-04-20 06:19:58,133 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:60000-0x12f72de7aee0000 Creating (or up
> dating) unassigned node for 1028785192 with OFFLINE state
> 2011-04-20 06:19:58,139 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=M_ZK_REGION_OFFLINE,
> server=HQSNOSQL01:60000, region=1028785192/.META.
> 2011-04-20 06:20:36,363 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
> out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305598133
> 2011-04-20 06:20:36,364 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for
> too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:20:36,364 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
> out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305598113
> 2011-04-20 06:20:36,364 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for
> too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:20:36,364 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
> was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305598133
> 2011-04-20 06:20:36,364 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
> was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305598113
> 2011-04-20 06:21:06,374 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
> out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305636364
> 2011-04-20 06:21:06,374 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for
> too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:21:06,374 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
> out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305636364
> 2011-04-20 06:21:06,374 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for
> too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:21:06,374 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
> was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305636364
> 2011-04-20 06:21:06,374 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
> was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305636364
> 2011-04-20 06:21:36,385 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
> out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305666374
> 2011-04-20 06:21:36,385 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for
> too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:21:36,385 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
> out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305666374
> 2011-04-20 06:21:36,385 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for
> too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:21:36,385 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
> was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305666374
> 2011-04-20 06:21:36,385 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
> was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305666374
> 2011-04-20 06:22:06,393 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
> out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305696385
> 2011-04-20 06:22:06,393 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for
> too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:22:06,393 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
> out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305696385
> 2011-04-20 06:22:06,393 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for
> too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:22:06,393 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
> was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305696385
> 2011-04-20 06:22:06,393 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
> was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305696385
> 2011-04-20 06:22:36,402 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
> out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305726393
> 2011-04-20 06:22:36,402 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for
> too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:22:36,402 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
> out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305726394
> 2011-04-20 06:22:36,402 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for
> too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:22:36,402 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
> was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305726393
> 2011-04-20 06:22:36,402 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
> was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305726394
> 2011-04-20 06:23:06,411 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
> out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305756402
> 2011-04-20 06:23:06,411 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for
> too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:23:06,411 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
> out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305756402
> 2011-04-20 06:23:06,411 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for
> too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:23:06,412 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
> was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305756402
> 2011-04-20 06:23:06,412 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
> was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305756402
> 2011-04-20 06:23:36,420 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
> out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305786412
> 2011-04-20 06:23:36,420 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for
> too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:23:36,420 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
> out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305786412
> 2011-04-20 06:23:36,420 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for
> too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:23:36,420 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
> was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305786412
> 2011-04-20 06:23:36,420 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
> was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305786412
> 2011-04-20 06:24:06,429 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
> out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305816420
> 2011-04-20 06:24:06,430 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for
> too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:24:06,430 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
> out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305816420
> 2011-04-20 06:24:06,430 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for
> too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:24:06,430 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
> was=.META.,,1.102878519
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
