[
https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13288941#comment-13288941
]
Enis Soztutar commented on HBASE-6160:
--------------------------------------
The exception:
{code}
12/06/04 06:50:41 ERROR security.UserGroupInformation:
PriviledgedActionException as:
cause:org.apache.hadoop.hbase.client.RegionOfflineException: Split daughter
region
TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d.
cannot be found in META.
Exception in thread "main"
org.apache.hadoop.hbase.client.RegionOfflineException: Split daughter region
TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d.
cannot be found in META.
at
org.apache.hadoop.hbase.client.MetaScanner$BlockingMetaScannerVisitor.processRow(MetaScanner.java:433)
at
org.apache.hadoop.hbase.client.MetaScanner$TableMetaScannerVisitor.processRow(MetaScanner.java:490)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:227)
at
org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:57)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:136)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:133)
at
org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:361)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:133)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:108)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:86)
at
org.apache.hadoop.hbase.client.MetaScanner.allTableRegions(MetaScanner.java:326)
at
org.apache.hadoop.hbase.client.HTable.getRegionLocations(HTable.java:499)
at
org.apache.hadoop.hbase.client.HTable.getStartEndKeys(HTable.java:452)
at
org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:132)
at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:962)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:979)
{code}
So the region in question is
{code}
50a4617eead34cad335a8dfa727d177d
{code}
and from the logs we see that {{25d9c4ff574a37bd95bf5e5be6d618dd}} is split
into {{1dc74065583c67b3916c4ed158cb53fa}} and
{{50a4617eead34cad335a8dfa727d177d}}
{code}
./hbase-hbase-regionserver-ip-10-226-65-102.log:2012-06-04 04:56:02,855 INFO
org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META updated,
and report to master.
Parent=TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd.,
new regions:
TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338800158687.1dc74065583c67b3916c4ed158cb53fa.,
TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d..
Split took 4sec
{code}
After some time, {{50a4617eead34cad335a8dfa727d177d}} is further split into
two:
{code}
./hbase-hbase-regionserver-ip-10-226-65-102.log:2012-06-04 05:41:13,488 INFO
org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META updated,
and report to master. Parent=TestLoadAndVerify_1
338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d.,
new regions:
TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338802866393.16288
65d7fa8e9eec3a7d8073465296e.,
TestLoadAndVerify_1338798130970,]y\x04\x00\x00\x00\x00\x00/000047_0,1338802866393.413cafe6c61426e26254c197e8c0a6ba..
Split took 7sec
{code}
Further time passes, and CatalogJanitor deletes the META entry for that region:
{code}
./hbase-hbase-master-ip-10-144-69-91.log:2012-06-04 05:47:16,688 DEBUG
org.apache.hadoop.hbase.master.CatalogJanitor: Deleting region
TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d.
because daughter splits no longer hold references
./hbase-hbase-master-ip-10-144-69-91.log:2012-06-04 05:47:18,103 INFO
org.apache.hadoop.hbase.catalog.MetaEditor: Deleted daughters references,
qualifier=splitA and qualifier=splitB, from parent
TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d.
./hbase-hbase-master-ip-10-144-69-91.log:2012-06-04 05:47:18,103 DEBUG
org.apache.hadoop.hbase.regionserver.HRegion: DELETING region
hdfs://ip-10-10-50-98.ec2.internal:8020/apps/hbase/data/TestLoadAndVerify_1338798130970/50a4617eead34cad335a8dfa727d177d
./hbase-hbase-master-ip-10-144-69-91.log:2012-06-04 05:47:18,145 INFO
org.apache.hadoop.hbase.catalog.MetaEditor: Deleted region
TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d.
from META
./hbase-hbase-master-ip-10-144-69-91.log:2012-06-04 05:52:12,891 WARN
org.apache.hadoop.hbase.master.CatalogJanitor: Daughter regiondir does not
exist:
hdfs://ip-10-10-50-98.ec2.internal:8020/apps/hbase/data/TestLoadAndVerify_1338798130970/50a4617eead34cad335a8dfa727d177d
{code}
However, there is no log for the grantfather,25d9c4ff574a37bd95bf5e5be6d618dd,
being deleted from META. In fact, META scan also confirms that the grandfather
is in META, but not the father:
{code}
TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd.
column=info:regioninfo, timestamp=1338800161105, value={NAME =>
'TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd.',
STARTKEY => '[\x02\x01\x00\x00\x00\x00\x00/000071_0', ENDKEY =>
'^P\xD7\x945\xE5\x0Dl', ENCODED => 25d9c4ff574a37bd95bf5e5be6d618dd, OFFLINE =>
true, SPLIT => true,}
TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd.
column=info:server, timestamp=1338799025521,
value=ip-10-226-65-102.ec2.internal:60020
TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd.
column=info:serverstartcode, timestamp=1338799025521, value=1338780117340
TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd.
column=info:splitA, timestamp=1338800161105, value={NAME =>
'TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338800158687.1dc74065583c67b3916c4ed158cb53fa.',
STARTKEY => '[\x02\x01\x00\x00\x00\x00\x00/000071_0', ENDKEY =>
'\\xA2\x04\x00\x00\x00\x00\x00/000048_0', ENCODED =>
1dc74065583c67b3916c4ed158cb53fa,}
TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd.
column=info:splitB, timestamp=1338800161105, value={NAME =>
'TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d.',
STARTKEY => '\\xA2\x04\x00\x00\x00\x00\x00/000048_0', ENDKEY =>
'^P\xD7\x945\xE5\x0Dl', ENCODED => 50a4617eead34cad335a8dfa727d177d,}
{code}
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the
> parent, but not the children. However, after the fix, we have seen the
> following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore,
> but META entry for Region A stays in META (C still refers it). In this case,
> the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
