stop hbase cluster clear the data of zookeeper start hbase cluster then the error would seem to disappeared
hope this to help someone otherwise there is a solution for somebody's problem : http://stackoverflow.com/questions/17792619/fatal-master-hmaster-unexpected-state-cannot-transit-it-to-offline On 28 November 2013 16:56, Jiajun Chen <[email protected]> wrote: > https://issues.apache.org/jira/browse/HBASE-8912 > > > On 28 November 2013 14:43, Jiajun Chen <[email protected]> wrote: > >> >> >> 2013-11-27 18:24:33,375 INFO >> org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter >> type for hdfs:// >> master.uc.uuc.com:9000/hbase/H/18c9cb11b3e673dec07038f166fb3ef7/.tmp/832ec249071c45b3934a186046ca429d: >> CompoundBloomFilterWriter >> 2013-11-27 18:24:33,385 INFO >> org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO >> DeleteFamily was added to HFile (hdfs:// >> master.uc.uuc.com:9000/hbase/H/18c9cb11b3e673dec07038f166fb3ef7/.tmp/832ec249071c45b3934a186046ca429d >> ) >> 2013-11-27 18:24:33,388 INFO org.apache.hadoop.hbase.regionserver.Store: >> Renaming compacted file at hdfs:// >> master.uc.uuc.com:9000/hbase/H/18c9cb11b3e673dec07038f166fb3ef7/.tmp/832ec249071c45b3934a186046ca429dto >> hdfs:// >> master.uc.uuc.com:9000/hbase/H/18c9cb11b3e673dec07038f166fb3ef7/page/832ec249071c45b3934a186046ca429d >> 2013-11-27 18:24:33,421 INFO org.apache.hadoop.hbase.regionserver.Store: >> Completed compaction of 3 file(s) in page of H, >> http://istock.jrj.com.cn/article,002024,6567377.html,1385541132079.18c9cb11b3e673dec07038f166fb3ef7. >> into 832ec249071c45b3934a186046ca429d, size=85.7k; total size for store is >> 533.2m >> 2013-11-27 18:24:33,422 INFO >> org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: >> completed compaction: regionName=H, >> http://istock.jrj.com.cn/article,002024,6567377.html,1385541132079.18c9cb11b3e673dec07038f166fb3ef7., >> storeName=page, fileCount=3, fileSize=91.8k, priority=3, >> time=1230511587222436; duration=0sec >> 2013-11-27 18:24:37,189 INFO >> org.apache.hadoop.hbase.regionserver.HRegionServer: >> regionserver60020.periodicFlusher requesting flush for region >> DocSortByHour,\x00\x00\x011\xAE\xCB\xD8 >> \x00\x00,B\x00\x00\x05\x84\x00\x01\xE6\xAF\x00\x02\x09\x0D\x00\x00\x00\x00\x06\xB5c\xB3,1314093149893.c5e2ce4602fb62714c72cd8da3b50bd5. >> after a delay of 5137 >> 2013-11-27 18:24:43,658 INFO org.apache.hadoop.hbase.util.FSUtils: >> FileSystem doesn't support getDefaultReplication >> 2013-11-27 18:24:43,658 INFO org.apache.hadoop.hbase.util.FSUtils: >> FileSystem doesn't support getDefaultBlockSize >> 2013-11-27 18:24:43,681 INFO >> org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter >> type for hdfs:// >> master.uc.uuc.com:9000/hbase/DocSortByHour/c5e2ce4602fb62714c72cd8da3b50bd5/.tmp/dd7c3908111d4fcc8945b2b6d5291364: >> CompoundBloomFilterWriter >> 2013-11-27 18:24:43,691 INFO >> org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO >> DeleteFamily was added to HFile (hdfs:// >> master.uc.uuc.com:9000/hbase/DocSortByHour/c5e2ce4602fb62714c72cd8da3b50bd5/.tmp/dd7c3908111d4fcc8945b2b6d5291364 >> ) >> 2013-11-27 18:24:43,691 INFO org.apache.hadoop.hbase.regionserver.Store: >> Flushed , sequenceid=26736748922, memsize=832.0, into tmp file hdfs:// >> master.uc.uuc.com:9000/hbase/DocSortByHour/c5e2ce4602fb62714c72cd8da3b50bd5/.tmp/dd7c3908111d4fcc8945b2b6d5291364 >> 2013-11-27 18:24:43,699 INFO org.apache.hadoop.hbase.regionserver.Store: >> Added hdfs:// >> master.uc.uuc.com:9000/hbase/DocSortByHour/c5e2ce4602fb62714c72cd8da3b50bd5/data/dd7c3908111d4fcc8945b2b6d5291364, >> entries=4, sequenceid=26736748922, filesize=1000.0 >> 2013-11-27 18:24:43,700 INFO >> org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of >> ~832.0/832, currentsize=0.0/0 for region >> DocSortByHour,\x00\x00\x011\xAE\xCB\xD8 >> \x00\x00,B\x00\x00\x05\x84\x00\x01\xE6\xAF\x00\x02\x09\x0D\x00\x00\x00\x00\x06\xB5c\xB3,1314093149893.c5e2ce4602fb62714c72cd8da3b50bd5. >> in 44ms, sequenceid=26736748922, compaction requested=true >> 2013-11-27 18:24:43,701 INFO >> org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on data >> in region DocSortByHour,\x00\x00\x011\xAE\xCB\xD8 >> \x00\x00,B\x00\x00\x05\x84\x00\x01\xE6\xAF\x00\x02\x09\x0D\x00\x00\x00\x00\x06\xB5c\xB3,1314093149893.c5e2ce4602fb62714c72cd8da3b50bd5. >> 2013-11-27 18:24:43,701 INFO org.apache.hadoop.hbase.regionserver.Store: >> Starting compaction of 3 file(s) in data of >> DocSortByHour,\x00\x00\x011\xAE\xCB\xD8 >> \x00\x00,B\x00\x00\x05\x84\x00\x01\xE6\xAF\x00\x02\x09\x0D\x00\x00\x00\x00\x06\xB5c\xB3,1314093149893.c5e2ce4602fb62714c72cd8da3b50bd5. >> into tmpdir=hdfs:// >> master.uc.uuc.com:9000/hbase/DocSortByHour/c5e2ce4602fb62714c72cd8da3b50bd5/.tmp, >> seqid=26736748922, totalSize=47.9m >> 2013-11-27 18:24:43,755 INFO org.apache.hadoop.hbase.util.FSUtils: >> FileSystem doesn't support getDefaultReplication >> 2013-11-27 18:24:43,756 INFO org.apache.hadoop.hbase.util.FSUtils: >> FileSystem doesn't support getDefaultBlockSize >> 2013-11-27 18:24:43,757 INFO >> org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter >> type for hdfs:// >> master.uc.uuc.com:9000/hbase/DocSortByHour/c5e2ce4602fb62714c72cd8da3b50bd5/.tmp/c2047a8a560645af936bcbcfe2625e5e: >> CompoundBloomFilterWriter >> 2013-11-27 18:24:47,354 INFO >> org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and >> DeleteFamily was added to HFile (hdfs:// >> master.uc.uuc.com:9000/hbase/DocSortByHour/c5e2ce4602fb62714c72cd8da3b50bd5/.tmp/c2047a8a560645af936bcbcfe2625e5e >> ) >> 2013-11-27 18:24:47,359 INFO >> org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded Delete Family >> Bloom (CompoundBloomFilter) metadata for c2047a8a560645af936bcbcfe2625e5e >> 2013-11-27 18:24:47,359 INFO org.apache.hadoop.hbase.regionserver.Store: >> Renaming compacted file at hdfs:// >> master.uc.uuc.com:9000/hbase/DocSortByHour/c5e2ce4602fb62714c72cd8da3b50bd5/.tmp/c2047a8a560645af936bcbcfe2625e5eto >> hdfs:// >> master.uc.uuc.com:9000/hbase/DocSortByHour/c5e2ce4602fb62714c72cd8da3b50bd5/data/c2047a8a560645af936bcbcfe2625e5e >> 2013-11-27 18:24:47,364 INFO >> org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded Delete Family >> Bloom (CompoundBloomFilter) metadata for c2047a8a560645af936bcbcfe2625e5e >> 2013-11-27 18:24:47,404 INFO org.apache.hadoop.hbase.regionserver.Store: >> Completed compaction of 3 file(s) in data of >> DocSortByHour,\x00\x00\x011\xAE\xCB\xD8 >> \x00\x00,B\x00\x00\x05\x84\x00\x01\xE6\xAF\x00\x02\x09\x0D\x00\x00\x00\x00\x06\xB5c\xB3,1314093149893.c5e2ce4602fb62714c72cd8da3b50bd5. >> into c2047a8a560645af936bcbcfe2625e5e, size=47.9m; total size for store is >> 223.8m >> 2013-11-27 18:24:47,404 INFO >> org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: >> completed compaction: regionName=DocSortByHour,\x00\x00\x011\xAE\xCB\xD8 >> \x00\x00,B\x00\x00\x05\x84\x00\x01\xE6\xAF\x00\x02\x09\x0D\x00\x00\x00\x00\x06\xB5c\xB3,1314093149893.c5e2ce4602fb62714c72cd8da3b50bd5., >> storeName=data, fileCount=3, fileSize=47.9m, priority=3, >> time=1230521954697044; duration=3sec >> 2013-11-27 18:27:07,191 INFO >> org.apache.hadoop.hbase.regionserver.HRegionServer: >> regionserver60020.periodicFlusher requesting flush for region >> h,!\xACU\xD2,1375542268855.ae4d6fae898f79e9fa019b78c975dd95. after a delay >> of 10647 >> 2013-11-27 18:27:17,190 INFO >> org.apache.hadoop.hbase.regionserver.HRegionServer: >> regionserver60020.periodicFlusher requesting flush for region >> h,!\xACU\xD2,1375542268855.ae4d6fae898f79e9fa019b78c975dd95. after a delay >> of 19328 >> 2013-11-27 18:27:17,841 INFO org.apache.hadoop.hbase.util.FSUtils: >> FileSystem doesn't support getDefaultReplication >> 2013-11-27 18:27:17,841 INFO org.apache.hadoop.hbase.util.FSUtils: >> FileSystem doesn't support getDefaultBlockSize >> 2013-11-27 18:27:17,851 INFO >> org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter >> type for hdfs:// >> master.uc.uuc.com:9000/hbase/h/ae4d6fae898f79e9fa019b78c975dd95/.tmp/135b6223aa9a41c1bf3511c7a389baea: >> CompoundBloomFilterWriter >> 2013-11-27 18:27:17,863 INFO >> org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO >> DeleteFamily was added to HFile (hdfs:// >> master.uc.uuc.com:9000/hbase/h/ae4d6fae898f79e9fa019b78c975dd95/.tmp/135b6223aa9a41c1bf3511c7a389baea >> ) >> >> >> >> On 28 November 2013 14:01, Ted Yu <[email protected]> wrote: >> >>> Jiajun: >>> Are you able to show us some more of the region server log ? >>> >>> Thanks >>> >>> On Nov 27, 2013, at 9:54 PM, Jiajun Chen <[email protected]> wrote: >>> >>> > I did't play with HBase 0.96+ at any point. >>> > >>> > and on d199.uuc.com >>> > >>> > 2013-11-27 18:24:33,421 INFO >>> org.apache.hadoop.hbase.regionserver.Store: >>> > Completed compaction of 3 file(s) in page of H, >>> > http://istock.jrj.com.cn/article,002024,6 >>> > 567377.html,1385541132079.18c9cb11b3e673dec07038f166fb3ef7. into >>> > 832ec249071c45b3934a186046ca429d, size=85.7k; total size for store is >>> 533.2m >>> > 2013-11-27 18:24:33,422 INFO >>> > org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: >>> > completed compaction: regionName=H,http://istock.jrj.com.cn/arti >>> > >>> cle,002024,6567377.html,1385541132079.18c9cb11b3e673dec07038f166fb3ef7., >>> > storeName=page, fileCount=3, fileSize=91.8k, priority=3, >>> > time=1230511587222436; duration >>> > =0sec >>> > >>> > >>> > >>> > >>> > On 28 November 2013 02:53, lars hofhansl <[email protected]> wrote: >>> > >>> >> Did you play with HBase 0.96+ at some point? Looks like there is junk >>> in a >>> >> znode. >>> >> >>> >> -- Lars >>> >> >>> >> >>> >> >>> >> ________________________________ >>> >> From: Jiajun Chen <[email protected]> >>> >> To: "[email protected]" <[email protected]> >>> >> Sent: Wednesday, November 27, 2013 3:21 AM >>> >> Subject: HMaster Aborted for HBase-0.94.13 >>> >> >>> >> >>> >> ...... >>> >> 2013-11-27 18:26:18,102 FATAL org.apache.hadoop.hbase.master.HMaster: >>> >> Master server abort: loaded coprocessors are: [] >>> >> 2013-11-27 18:26:18,102 FATAL org.apache.hadoop.hbase.master.HMaster: >>> >> Unexpected state : H, >>> >> >>> >> >>> http://istock.jrj.com.cn/article,002024,6567377.html,1385541132079.18c9cb11b3e673dec07038f166fb3ef7 >>> . >>> >> state=PENDING_O >>> >> PEN, ts=1385547978102, server=d199.uuc.com,60020,1385047501649 .. >>> Cannot >>> >> transit it to OFFLINE. >>> >> java.lang.IllegalStateException: Unexpected state : H, >>> >> >>> >> >>> http://istock.jrj.com.cn/article,002024,6567377.html,1385541132079.18c9cb11b3e673dec07038f166fb3ef7 >>> >> . >>> >> state=PENDING_OPEN, ts=1385547978102, server=d199.uuc.com >>> >> ,60020,1385047501649 >>> >> .. Cannot transit it to OFFLINE. >>> >> at >>> >> >>> >> >>> org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1890) >>> >> at >>> >> >>> >> >>> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1690) >>> >> at >>> >> >>> >> >>> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1426) >>> >> at >>> >> >>> >> >>> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1398) >>> >> at >>> >> >>> >> >>> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1393) >>> >> at >>> >> >>> >> >>> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105) >>> >> at >>> >> >>> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175) >>> >> at >>> >> >>> >> >>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>> >> at >>> >> >>> >> >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>> >> at java.lang.Thread.run(Thread.java:662) >>> >> 2013-11-27 18:26:18,104 INFO org.apache.hadoop.hbase.master.HMaster: >>> >> Aborting >>> >> 2013-11-27 18:26:18,104 INFO org.apache.hadoop.ipc.HBaseServer: >>> Stopping >>> >> server on 60000 >>> >> 2013-11-27 18:26:18,104 INFO org.apache.hadoop.ipc.HBaseServer: IPC >>> Server >>> >> handler 1 on 60000: exiting >>> >> ...... >>> >> and >>> >> It worked well If I used version 0.94.3 . >>> >> >>> >> >> >
