[
https://issues.apache.org/jira/browse/HBASE-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895726#action_12895726
]
Dave Latham commented on HBASE-2812:
------------------------------------
I have also run into this, using 0.20.6. I have a test environment that runs a
set of automated tests with our latest code every hour. Some of these tests
are set to disable, drop, and recreate a table in order to make an empty one.
After succeeding for 5 to 10 executions, it seems to get into this state. I'm
not able to disable and get rid of the test table until hbase is restarted, so
it is definitely hampering our ability to do automated testing with hbase.
The logs I have look similar, though the master and region server are separate
processes with separate logs in my environment.
Does anyone have any ideas as to the cause of this? Is there any more
information that would help in diagnosing?
> Disable 'table' fails to complete frustrating my ability to test easily
> -----------------------------------------------------------------------
>
> Key: HBASE-2812
> URL: https://issues.apache.org/jira/browse/HBASE-2812
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.20.6, 0.89.20100621
> Environment: 0.89, non-distributed mode
> Reporter: Sam Pullara
> Fix For: 0.20.7, 0.90.0
>
>
> I see this in the client after it gives up:
> hbase(main):006:0> disable 'test_schema'
> ERROR: org.apache.hadoop.hbase.RegionException: Retries exhausted, it took
> too long to wait for the table test_schema to be disabled.
> Here is some help for this command:
> Disable the named table: e.g. "hbase> disable 't1'"
> and this in the server log, a set of about 5 reports it is closing per
> disable call:
> 2010-07-03 15:19:47,554 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Processing unserved regions
> 2010-07-03 15:19:47,554 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Processing regions currently
> being served
> 2010-07-03 15:19:47,555 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Adding region
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081. to setClosing
> list
> 2010-07-03 15:19:47,576 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:47,576 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_CLOSE:
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:48,567 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Processing unserved regions
> 2010-07-03 15:19:48,567 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Processing regions currently
> being served
> 2010-07-03 15:19:48,568 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Adding region
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081. to setClosing
> list
> 2010-07-03 15:19:48,577 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:48,578 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_CLOSE:
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:49,580 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Processing unserved regions
> 2010-07-03 15:19:49,580 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Processing regions currently
> being served
> 2010-07-03 15:19:49,581 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Adding region
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081. to setClosing
> list
> 2010-07-03 15:19:50,580 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:50,581 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_CLOSE:
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:50,592 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Processing unserved regions
> 2010-07-03 15:19:50,592 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Processing regions currently
> being served
> 2010-07-03 15:19:50,593 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Adding region
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081. to setClosing
> list
> 2010-07-03 15:19:51,581 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:51,581 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_CLOSE:
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:52,605 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Processing unserved regions
> 2010-07-03 15:19:52,605 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Processing regions currently
> being served
> 2010-07-03 15:19:52,606 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Adding region
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081. to setClosing
> list
> 2010-07-03 15:19:52,703 INFO org.apache.hadoop.hbase.master.ServerManager: 1
> region servers, 0 dead, average load 3.0
> 2010-07-03 15:19:52,863 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scanning meta region {server: 192.168.2.1:54389,
> regionname: -ROOT-,,0.70236052, startKey: <>}
> 2010-07-03 15:19:52,867 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scan of 1 row(s) of meta region {server:
> 192.168.2.1:54389, regionname: -ROOT-,,0.70236052, startKey: <>} complete
> 2010-07-03 15:19:53,585 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:53,585 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_CLOSE:
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:53,671 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache:
> Cache Stats: Sizes: Total=1.6292953MB (1708440), Free=196.70822MB
> (206263512), Max=198.33751MB (207971952), Counts: Blocks=2, Access=52,
> Hit=50, Miss=2, Evictions=0, Evicted=0, Ratios: Hit Ratio=96.15384340286255%,
> Miss Ratio=3.8461539894342422%, Evicted/Run=NaN
> 2010-07-03 15:19:54,617 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Processing unserved regions
> 2010-07-03 15:19:54,617 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Processing regions currently
> being served
> 2010-07-03 15:19:54,618 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Adding region
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081. to setClosing
> list
> 2010-07-03 15:19:54,821 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scanning meta region {server: 192.168.2.1:54389,
> regionname: .META.,,1.1028785192, startKey: <>}
> 2010-07-03 15:19:54,825 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scan of 2 row(s) of meta region {server:
> 192.168.2.1:54389, regionname: .META.,,1.1028785192, startKey: <>} complete
> 2010-07-03 15:19:54,825 INFO org.apache.hadoop.hbase.master.BaseScanner: All
> 1 .META. region(s) scanned
> 2010-07-03 15:19:55,589 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:55,589 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_CLOSE:
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:58,629 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Processing unserved regions
> 2010-07-03 15:19:58,629 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Processing regions currently
> being served
> 2010-07-03 15:19:58,630 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Adding region
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081. to setClosing
> list
> 2010-07-03 15:19:59,595 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:59,595 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_CLOSE:
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:20:52,706 INFO org.apache.hadoop.hbase.master.ServerManager: 1
> region servers, 0 dead, average load 3.0
> 2010-07-03 15:20:52,866 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scanning meta region {server: 192.168.2.1:54389,
> regionname: -ROOT-,,0.70236052, startKey: <>}
> 2010-07-03 15:20:52,873 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scan of 1 row(s) of meta region {server:
> 192.168.2.1:54389, regionname: -ROOT-,,0.70236052, startKey: <>} complete
> 2010-07-03 15:20:53,671 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache:
> Cache Stats: Sizes: Total=1.6292953MB (1708440), Free=196.70822MB
> (206263512), Max=198.33751MB (207971952), Counts: Blocks=2, Access=54,
> Hit=52, Miss=2, Evictions=0, Evicted=0, Ratios: Hit Ratio=96.29629850387573%,
> Miss Ratio=3.7037037312984467%, Evicted/Run=NaN
> 2010-07-03 15:20:54,823 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scanning meta region {server: 192.168.2.1:54389,
> regionname: .META.,,1.1028785192, startKey: <>}
> 2010-07-03 15:20:54,828 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scan of 2 row(s) of meta region {server:
> 192.168.2.1:54389, regionname: .META.,,1.1028785192, startKey: <>} complete
> 2010-07-03 15:20:54,828 INFO org.apache.hadoop.hbase.master.BaseScanner: All
> 1 .META. region(s) scanned
> 2010-07-03 15:21:52,709 INFO org.apache.hadoop.hbase.master.ServerManager: 1
> region servers, 0 dead, average load 3.0
> 2010-07-03 15:21:52,869 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scanning meta region {server: 192.168.2.1:54389,
> regionname: -ROOT-,,0.70236052, startKey: <>}
> 2010-07-03 15:21:52,876 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scan of 1 row(s) of meta region {server:
> 192.168.2.1:54389, regionname: -ROOT-,,0.70236052, startKey: <>} complete
> 2010-07-03 15:21:53,672 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache:
> Cache Stats: Sizes: Total=1.6292953MB (1708440), Free=196.70822MB
> (206263512), Max=198.33751MB (207971952), Counts: Blocks=2, Access=56,
> Hit=54, Miss=2, Evictions=0, Evicted=0, Ratios: Hit Ratio=96.42857313156128%,
> Miss Ratio=3.57142873108387%, Evicted/Run=NaN
> 2010-07-03 15:21:54,826 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scanning meta region {server: 192.168.2.1:54389,
> regionname: .META.,,1.1028785192, startKey: <>}
> 2010-07-03 15:21:54,830 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scan of 2 row(s) of meta region {server:
> 192.168.2.1:54389, regionname: .META.,,1.1028785192, startKey: <>} complete
> 2010-07-03 15:21:54,830 INFO org.apache.hadoop.hbase.master.BaseScanner: All
> 1 .META. region(s) scanned
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
