On 03/10/2010 07:58 PM, Jean-Daniel Cryans wrote:
Which HBase version? What's your hardware like? How much data were you
inserting? Did you grep the region server logs for any IOException or
such? Can we see an excerpt of those logs around the time of the "lock
up"?
Version: 0.20.3-1.cloudera
Hardware: dual Xeon 4 core, 16G, 1.7T disk
10x nodes: 1 master, 1 secondary master, 8x regionservers. 2x
zookeepers running on regionservers
It appears to have died after only a few rows were inserted. There's
only one region shown on the status page. Curiously, that region does
NOT show up in the list of online regions for the listed regionserver.
Master log, from the point where I ran "drop 'Person'" in the shell:
010-03-10 20:44:44,812 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scanning meta region {server: 10.40.0.37:60020, regionname:
-ROOT-,,0, startKey:<>}
2010-03-10 20:44:44,815 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scan of 1 row(s) of meta region {server: 10.40.0.37:60020,
regionname: -ROOT-,,0, startKey:<>} complete
2010-03-10 20:44:44,836 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {server: 10.40.0.36:60020, regionname:
.META.,,1, startKey:<>}
2010-03-10 20:44:44,844 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scan of 3 row(s) of meta region {server: 10.40.0.36:60020,
regionname: .META.,,1, startKey:<>} complete
2010-03-10 20:44:44,844 INFO org.apache.hadoop.hbase.master.BaseScanner: All 1
.META. region(s) scanned
2010-03-10 20:44:45,357 INFO org.apache.hadoop.hbase.master.ServerManager: 5
region servers, 0 dead, average load 1.2
2010-03-10 20:45:03,209 DEBUG org.apache.hadoop.hbase.master.ChangeTableState:
Processing unserved regions
2010-03-10 20:45:03,209 DEBUG org.apache.hadoop.hbase.master.ChangeTableState:
Processing regions currently being served
2010-03-10 20:45:03,210 DEBUG org.apache.hadoop.hbase.master.ChangeTableState:
Adding region Person,,1268251509658 to setClosing list
2010-03-10 20:45:04,260 DEBUG org.apache.hadoop.hbase.master.ChangeTableState:
Processing unserved regions
2010-03-10 20:45:04,260 DEBUG org.apache.hadoop.hbase.master.ChangeTableState:
Processing regions currently being served
2010-03-10 20:45:04,260 DEBUG org.apache.hadoop.hbase.master.ChangeTableState:
Adding region Person,,1268251509658 to setClosing list
2010-03-10 20:45:05,273 DEBUG org.apache.hadoop.hbase.master.ChangeTableState:
Processing unserved regions
2010-03-10 20:45:05,273 DEBUG org.apache.hadoop.hbase.master.ChangeTableState:
Processing regions currently being served
2010-03-10 20:45:05,273 DEBUG org.apache.hadoop.hbase.master.ChangeTableState:
Adding region Person,,1268251509658 to setClosing list
2010-03-10 20:45:06,287 DEBUG org.apache.hadoop.hbase.master.ChangeTableState:
Processing unserved regions
2010-03-10 20:45:06,287 DEBUG org.apache.hadoop.hbase.master.ChangeTableState:
Processing regions currently being served
2010-03-10 20:45:06,287 DEBUG org.apache.hadoop.hbase.master.ChangeTableState:
Adding region Person,,1268251509658 to setClosing list
2010-03-10 20:45:08,301 DEBUG org.apache.hadoop.hbase.master.ChangeTableState:
Processing unserved regions
2010-03-10 20:45:08,301 DEBUG org.apache.hadoop.hbase.master.ChangeTableState:
Processing regions currently being served
2010-03-10 20:45:08,301 DEBUG org.apache.hadoop.hbase.master.ChangeTableState:
Adding region Person,,1268251509658 to setClosing list
Log from the region server where the region is supposed to be for the
same time frame:
2010-03-10 20:43:50,889 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache:
Cache Stats: Sizes: Total=1.6213074MB (1700064), Free=195.8787MB (205393696),
Max=197.5MB (207093760), Counts: Blocks=0, Access=0, Hit=0, Miss=0,
Evictions=0, Evicted=0, Ratios: Hit Ratio=NaN%, Miss Ratio=NaN%, Evicted/Run=NaN
2010-03-10 20:44:50,889 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache:
Cache Stats: Sizes: Total=1.6213074MB (1700064), Free=195.8787MB (205393696),
Max=197.5MB (207093760), Counts: Blocks=0, Access=0, Hit=0, Miss=0,
Evictions=0, Evicted=0, Ratios: Hit Ratio=NaN%, Miss Ratio=NaN%, Evicted/Run=NaN
2010-03-10 20:45:04,058 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
Person,,1268251509658
2010-03-10 20:45:04,059 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_CLOSE:
Person,,1268251509658
2010-03-10 20:45:05,062 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
Person,,1268251509658
2010-03-10 20:45:05,063 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_CLOSE:
Person,,1268251509658
2010-03-10 20:45:06,066 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
Person,,1268251509658
2010-03-10 20:45:06,067 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_CLOSE:
Person,,1268251509658
2010-03-10 20:45:07,070 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
Person,,1268251509658
2010-03-10 20:45:07,071 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_CLOSE:
Person,,1268251509658
2010-03-10 20:45:09,079 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
Person,,1268251509658
2010-03-10 20:45:09,079 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_CLOSE:
Person,,1268251509658
2010-03-10 20:45:11,088 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
Person,,1268251509658
2010-03-10 20:45:11,088 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_CLOSE:
Person,,1268251509658
2010-03-10 20:45:15,104 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
Person,,1268251509658
2010-03-10 20:45:15,105 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_CLOSE:
Person,,1268251509658
2010-03-10 20:45:50,889 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache:
Cache Stats: Sizes: Total=1.6213074MB (1700064), Free=195.8787MB (205393696),
Max=197.5MB (207093760), Counts: Blocks=0, Access=0, Hit=0, Miss=0,
Evictions=0, Evicted=0, Ratios: Hit Ratio=NaN%, Miss Ratio=NaN%, Evicted/Run=NaN
