We were runing the HBase cluster for one whole night.
Then we tried to restart the region server. That we got an error saying transition of a region failed due to version mismatch in the unassigned node. The META region was actually getting loaded as part of log replay that is when we got the transition error. The surprising part was that the difference in mismatch was sometimes 1 or sometimes 2 for the same Region. 2011-04-14 11:07:19,786 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=106.56 KB, free=12.28 MB, max=12.39 MB, blocks=2, accesses=2, hits=0, hitRatio=0.00%%, cachingAccesses=2, cachingHits=0, cachingHitsRatio=0.00%%, evictions=0, evicted=0, evictedPerRun=NaN 2011-04-14 11:07:35,339 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x12f52766dcd0001 Attempting to transition node 1028785192/.META. from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2011-04-14 11:07:35,350 WARN org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x12f52766dcd0001 Attempt to transition the unassigned node for 1028785192 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING failed, the node existed but was version 2 not the expected version 1 2011-04-14 11:07:35,350 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed refreshing OPENING; region=1028785192, context=open_region_progress 2011-04-14 11:07:35,350 WARN org.apache.hadoop.hbase.regionserver.HRegion: Progressable reporter failed, stopping replay 2011-04-14 11:07:35,386 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of region=.META.,,1.1028785192 regionserver:60020-0x12f52766dcd0001 Attempting to transition node 1028785192/.META. from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2011-04-14 11:08:33,744 WARN org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x12f52766dcd0001 Attempt to transition the unassigned node for 1028785192 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING failed, the node existed but was version 5 not the expected version 3 2011-04-14 11:08:33,744 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed refreshing OPENING; region=1028785192, context=open_region_progress 2011-04-14 11:08:33,744 WARN org.apache.hadoop.hbase.regionserver.HRegion: Progressable reporter failed, stopping replay 2011-04-14 11:08:33,775 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of region=.META.,,1.1028785192 **************************************************************************** *********** This e-mail and attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient's) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!
