[
https://issues.apache.org/jira/browse/HBASE-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895749#action_12895749
]
Jean-Daniel Cryans commented on HBASE-2812:
-------------------------------------------
Dave sent me the logs, and it really looks like a case of HBASE-2755:
{noformat}
2010-08-04 19:26:58,291 DEBUG org.apache.hadoop.hbase.master.ChangeTableState:
Adding region test_ReferralQueue,,1280975190155 to setClosing list
2010-08-04 19:26:58,565 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {server: 192.168.21.66:60020,
regionname: .META.,,1, startKey: <>}
2010-08-04 19:26:58,654 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_CLOSE: test_ReferralQueue,,1280975190155 from
hslave3,60020,1280951333279; 1 of 1
2010-08-04 19:26:58,654 DEBUG org.apache.hadoop.hbase.master.HMaster:
Processing todo: ProcessRegionClose of test_ReferralQueue,,1280975190155, true,
reassign: false
2010-08-04 19:26:58,656 INFO
org.apache.hadoop.hbase.master.ProcessRegionClose$1: region closed:
test_ReferralQueue,,1280975190155
2010-08-04 19:26:58,686 DEBUG org.apache.hadoop.hbase.master.BaseScanner: GET
on test_ReferralQueue,,1280975190155 got different startcode than SCAN: sc=0,
serverAddress=1280951333279
2010-08-04 19:26:58,686 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
Current assignment of test_ReferralQueue,,1280975190155 is not valid;
serverAddress=, startCode=0 unknown.
{noformat}
Here the base scanner saw that that region was half unassigned (atomicity
problem?) since we don't see the message that "sa" is different. It means
that the base scanner reassigns the region because it believes that:
- the region isn't assigned to anyone
- that it's not online
- that it's not in transition
This means that we should also verify our assumption that the HRI.isOffline()
is correct when we read it. I think we can augment HBASE-2755's patch to do
that, since doing the GET we already read the new HRI. Making a patch.
> Disable 'table' fails to complete frustrating my ability to test easily
> -----------------------------------------------------------------------
>
> Key: HBASE-2812
> URL: https://issues.apache.org/jira/browse/HBASE-2812
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.20.6, 0.89.20100621
> Environment: 0.89, non-distributed mode
> Reporter: Sam Pullara
> Fix For: 0.20.7, 0.90.0
>
>
> I see this in the client after it gives up:
> hbase(main):006:0> disable 'test_schema'
> ERROR: org.apache.hadoop.hbase.RegionException: Retries exhausted, it took
> too long to wait for the table test_schema to be disabled.
> Here is some help for this command:
> Disable the named table: e.g. "hbase> disable 't1'"
> and this in the server log, a set of about 5 reports it is closing per
> disable call:
> 2010-07-03 15:19:47,554 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Processing unserved regions
> 2010-07-03 15:19:47,554 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Processing regions currently
> being served
> 2010-07-03 15:19:47,555 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Adding region
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081. to setClosing
> list
> 2010-07-03 15:19:47,576 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:47,576 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_CLOSE:
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:48,567 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Processing unserved regions
> 2010-07-03 15:19:48,567 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Processing regions currently
> being served
> 2010-07-03 15:19:48,568 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Adding region
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081. to setClosing
> list
> 2010-07-03 15:19:48,577 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:48,578 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_CLOSE:
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:49,580 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Processing unserved regions
> 2010-07-03 15:19:49,580 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Processing regions currently
> being served
> 2010-07-03 15:19:49,581 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Adding region
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081. to setClosing
> list
> 2010-07-03 15:19:50,580 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:50,581 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_CLOSE:
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:50,592 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Processing unserved regions
> 2010-07-03 15:19:50,592 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Processing regions currently
> being served
> 2010-07-03 15:19:50,593 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Adding region
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081. to setClosing
> list
> 2010-07-03 15:19:51,581 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:51,581 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_CLOSE:
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:52,605 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Processing unserved regions
> 2010-07-03 15:19:52,605 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Processing regions currently
> being served
> 2010-07-03 15:19:52,606 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Adding region
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081. to setClosing
> list
> 2010-07-03 15:19:52,703 INFO org.apache.hadoop.hbase.master.ServerManager: 1
> region servers, 0 dead, average load 3.0
> 2010-07-03 15:19:52,863 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scanning meta region {server: 192.168.2.1:54389,
> regionname: -ROOT-,,0.70236052, startKey: <>}
> 2010-07-03 15:19:52,867 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scan of 1 row(s) of meta region {server:
> 192.168.2.1:54389, regionname: -ROOT-,,0.70236052, startKey: <>} complete
> 2010-07-03 15:19:53,585 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:53,585 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_CLOSE:
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:53,671 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache:
> Cache Stats: Sizes: Total=1.6292953MB (1708440), Free=196.70822MB
> (206263512), Max=198.33751MB (207971952), Counts: Blocks=2, Access=52,
> Hit=50, Miss=2, Evictions=0, Evicted=0, Ratios: Hit Ratio=96.15384340286255%,
> Miss Ratio=3.8461539894342422%, Evicted/Run=NaN
> 2010-07-03 15:19:54,617 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Processing unserved regions
> 2010-07-03 15:19:54,617 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Processing regions currently
> being served
> 2010-07-03 15:19:54,618 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Adding region
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081. to setClosing
> list
> 2010-07-03 15:19:54,821 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scanning meta region {server: 192.168.2.1:54389,
> regionname: .META.,,1.1028785192, startKey: <>}
> 2010-07-03 15:19:54,825 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scan of 2 row(s) of meta region {server:
> 192.168.2.1:54389, regionname: .META.,,1.1028785192, startKey: <>} complete
> 2010-07-03 15:19:54,825 INFO org.apache.hadoop.hbase.master.BaseScanner: All
> 1 .META. region(s) scanned
> 2010-07-03 15:19:55,589 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:55,589 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_CLOSE:
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:58,629 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Processing unserved regions
> 2010-07-03 15:19:58,629 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Processing regions currently
> being served
> 2010-07-03 15:19:58,630 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Adding region
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081. to setClosing
> list
> 2010-07-03 15:19:59,595 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:59,595 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_CLOSE:
> test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:20:52,706 INFO org.apache.hadoop.hbase.master.ServerManager: 1
> region servers, 0 dead, average load 3.0
> 2010-07-03 15:20:52,866 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scanning meta region {server: 192.168.2.1:54389,
> regionname: -ROOT-,,0.70236052, startKey: <>}
> 2010-07-03 15:20:52,873 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scan of 1 row(s) of meta region {server:
> 192.168.2.1:54389, regionname: -ROOT-,,0.70236052, startKey: <>} complete
> 2010-07-03 15:20:53,671 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache:
> Cache Stats: Sizes: Total=1.6292953MB (1708440), Free=196.70822MB
> (206263512), Max=198.33751MB (207971952), Counts: Blocks=2, Access=54,
> Hit=52, Miss=2, Evictions=0, Evicted=0, Ratios: Hit Ratio=96.29629850387573%,
> Miss Ratio=3.7037037312984467%, Evicted/Run=NaN
> 2010-07-03 15:20:54,823 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scanning meta region {server: 192.168.2.1:54389,
> regionname: .META.,,1.1028785192, startKey: <>}
> 2010-07-03 15:20:54,828 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scan of 2 row(s) of meta region {server:
> 192.168.2.1:54389, regionname: .META.,,1.1028785192, startKey: <>} complete
> 2010-07-03 15:20:54,828 INFO org.apache.hadoop.hbase.master.BaseScanner: All
> 1 .META. region(s) scanned
> 2010-07-03 15:21:52,709 INFO org.apache.hadoop.hbase.master.ServerManager: 1
> region servers, 0 dead, average load 3.0
> 2010-07-03 15:21:52,869 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scanning meta region {server: 192.168.2.1:54389,
> regionname: -ROOT-,,0.70236052, startKey: <>}
> 2010-07-03 15:21:52,876 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scan of 1 row(s) of meta region {server:
> 192.168.2.1:54389, regionname: -ROOT-,,0.70236052, startKey: <>} complete
> 2010-07-03 15:21:53,672 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache:
> Cache Stats: Sizes: Total=1.6292953MB (1708440), Free=196.70822MB
> (206263512), Max=198.33751MB (207971952), Counts: Blocks=2, Access=56,
> Hit=54, Miss=2, Evictions=0, Evicted=0, Ratios: Hit Ratio=96.42857313156128%,
> Miss Ratio=3.57142873108387%, Evicted/Run=NaN
> 2010-07-03 15:21:54,826 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scanning meta region {server: 192.168.2.1:54389,
> regionname: .META.,,1.1028785192, startKey: <>}
> 2010-07-03 15:21:54,830 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scan of 2 row(s) of meta region {server:
> 192.168.2.1:54389, regionname: .META.,,1.1028785192, startKey: <>} complete
> 2010-07-03 15:21:54,830 INFO org.apache.hadoop.hbase.master.BaseScanner: All
> 1 .META. region(s) scanned
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
