[
https://issues.apache.org/jira/browse/HBASE-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12573598#action_12573598
]
stack commented on HBASE-471:
-----------------------------
Lars let me up on his cluster. His tables are riddled with hbase-428 issues:
i.e. he has loads of regions that start with and end with same key. There is
one item I cannot currently explain and thats the fact that the table has two
regions that both start with the null key. Should never happen. DEBUG logging
was enabled after creation so can't see how the scenario came about. Strangely
things seem to be sort of working in that the enveloping region -- the start
region that spans a larger key space -- seems to be splitting frequently either
because load is still on or because its trying to get the region back under the
upper size bound.
Plan is to get Lars up on a cluster with the hbase428 patch and then make a
FixHBase428 tool to run against his table to do fixup. Will try and leverage
merge regions tool. For every instance of a region found where start and end
key are same AND a subsequent region starts with same key, will try and merge
these two (Unfortunately the two regions with same start and end key actually
have data -- its all data of the start row -- so can't just delete this
region). Will also try merge of the first two regions.
> IllegalStateException thrown in client after region was split and deleted
> -------------------------------------------------------------------------
>
> Key: HBASE-471
> URL: https://issues.apache.org/jira/browse/HBASE-471
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.1.0
> Environment: Linux Debian, HBase 0.16.0
> Reporter: Lars George
> Attachments: hbase-master-log.tar.gz, logs.tar.gz
>
>
> For some reason a client sometimes fails to locate a row with a
> IllegalStateException when the region was split and deleted.
> > [2008-02-25 16:12:39,171] ERROR [http-80-Processor20]
> > archive.MultilingualArchive - getDocument: An error occurred.
> > java.lang.IllegalStateException: region offline:
> > pdc-docs,US7039976_20060509,1203981958556
> > at
> > org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:432)
> > at
> > org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:343)
> > at
> > org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:306)
> > at org.apache.hadoop.hbase.HTable.getRegionLocation(HTable.java:102)
> > at org.apache.hadoop.hbase.HTable.get(HTable.java:280)
> Tracing the region on the master shows this:
> > 2008-02-25 16:09:38,761 DEBUG org.apache.hadoop.hbase.HMaster: Received
> > MSG_REGION_SPLIT :
> +pdc-docs,US7039976_20060509,1203981958556 from 192.168.105.21:60020
> > 2008-02-25 16:09:38,761 INFO org.apache.hadoop.hbase.HMaster: region
> > pdc-docs,US7039976_20060509,1203981958556 split. New regions
> +are: pdc-docs,US7039976_20060509,1203984578345,
> pdc-docs,US7046359_20060516,1203984578345
> > 2008-02-25 16:10:02,470 DEBUG org.apache.hadoop.hbase.HMaster:
> > HMaster.metaScanner regioninfo: {regionname:
> +pdc-docs,US7039976_20060509,1203981958556, startKey: <US7039976_20060509>,
> endKey: <US7053021_20060530>, encodedName: 1260314009,
> +offline: true, split: true, tableDesc: {name: pdc-docs, families:
> {contents:={name: contents, max versions: 3, compression: NONE, in
> +memory: false, max length: 2147483647, bloom filter: none}, language:={name:
> language, max versions: 3, compression: NONE, in
> +memory: false, max length: 2147483647, bloom filter: none}, mimetype:={name:
> mimetype, max versions: 3, compression: NONE, in
> +memory: false, max length: 2147483647, bloom filter: none}}}}, server:
> 192.168.105.21:60020, startCode: 1203949130468
> > 2008-02-25 16:10:02,513 DEBUG org.apache.hadoop.hbase.HMaster:
> > pdc-docs,US7039976_20060509,1203984578345 no longer has references
> +to pdc-docs,US7039976_20060509,1203981958556
> > 2008-02-25 16:10:02,516 DEBUG org.apache.hadoop.hbase.HMaster:
> > pdc-docs,US7046359_20060516,1203984578345 no longer has references
> +to pdc-docs,US7039976_20060509,1203981958556
> > 2008-02-25 16:10:02,516 INFO org.apache.hadoop.hbase.HMaster: Deleting
> > region pdc-docs,US7039976_20060509,1203981958556 because
> +daughter splits no longer hold references
> After discussion with st^ack it seems that the server is simply not retrying
> IllegalStateException but IOExceptions only. Also see HBASE-452 which should
> be addressed at the same time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
