[
https://issues.apache.org/jira/browse/HBASE-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12572631#action_12572631
]
Lars George commented on HBASE-471:
-----------------------------------
I do not have those but it just happened to my in HQL, this time I grepped it:
hql > select * from pdc-trans;
Exception in thread "main" java.lang.IllegalStateException: region offline:
pdc-docs,US20040151310_20040805,1204047862746
at
org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:432)
at
org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:343)
at
org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:306)
at org.apache.hadoop.hbase.HTable.<init>(HTable.java:90)
at
org.apache.hadoop.hbase.hql.SelectCommand.execute(SelectCommand.java:90)
at org.apache.hadoop.hbase.hql.HQLClient.executeQuery(HQLClient.java:50)
at org.apache.hadoop.hbase.Shell.main(Shell.java:114)
>From the Master:
2008-02-26 11:38:06,453 DEBUG org.apache.hadoop.hbase.HMaster:
HMaster.metaScanner regioninfo: {regionname:
pdc-docs,US20040151310_20040805,1204047862746, startKey:
<US20040151310_20040805>, endKey: <US20040157377_20040812>, encodedName:
516792513, tableDesc: {name: pdc-docs, families: {contents:={name: contents,
max versions: 3, compression: NONE, in memory: false, max length: 2147483647,
bloom filter: none}, language:={name: language, max versions: 3, compression:
NONE, in memory: false, max length: 2147483647, bloom filter: none},
mimetype:={name: mimetype, max versions: 3, compression: NONE, in memory:
false, max length: 2147483647, bloom filter: none}}}}, server:
192.168.105.29:60020, startCode: 1203949130420
2008-02-26 11:39:06,453 DEBUG org.apache.hadoop.hbase.HMaster:
HMaster.metaScanner regioninfo: {regionname:
pdc-docs,US20040151310_20040805,1204047862746, startKey:
<US20040151310_20040805>, endKey: <US20040157377_20040812>, encodedName:
516792513, tableDesc: {name: pdc-docs, families: {contents:={name: contents,
max versions: 3, compression: NONE, in memory: false, max length: 2147483647,
bloom filter: none}, language:={name: language, max versions: 3, compression:
NONE, in memory: false, max length: 2147483647, bloom filter: none},
mimetype:={name: mimetype, max versions: 3, compression: NONE, in memory:
false, max length: 2147483647, bloom filter: none}}}}, server:
192.168.105.29:60020, startCode: 1203949130420
2008-02-26 11:39:53,733 DEBUG org.apache.hadoop.hbase.HMaster: Received
MSG_REGION_SPLIT : pdc-docs,US20040151310_20040805,1204047862746 from
192.168.105.29:60020
2008-02-26 11:39:53,733 INFO org.apache.hadoop.hbase.HMaster: region
pdc-docs,US20040151310_20040805,1204047862746 split. New regions are:
pdc-docs,US20040151310_20040805,1204054791249,
pdc-docs,US20040154199_20040812,1204054791249
2008-02-26 11:40:06,452 DEBUG org.apache.hadoop.hbase.HMaster:
HMaster.metaScanner regioninfo: {regionname:
pdc-docs,US20040151310_20040805,1204047862746, startKey:
<US20040151310_20040805>, endKey: <US20040157377_20040812>, encodedName:
516792513, offline: true, split: true, tableDesc: {name: pdc-docs, families:
{contents:={name: contents, max versions: 3, compression: NONE, in memory:
false, max length: 2147483647, bloom filter: none}, language:={name: language,
max versions: 3, compression: NONE, in memory: false, max length: 2147483647,
bloom filter: none}, mimetype:={name: mimetype, max versions: 3, compression:
NONE, in memory: false, max length: 2147483647, bloom filter: none}}}}, server:
192.168.105.29:60020, startCode: 1203949130420
2008-02-26 11:40:06,634 DEBUG org.apache.hadoop.hbase.HMaster:
pdc-docs,US20040151310_20040805,1204054791249 no longer has references to
pdc-docs,US20040151310_20040805,1204047862746
2008-02-26 11:40:06,636 DEBUG org.apache.hadoop.hbase.HMaster: Checked
pdc-docs,US20040151310_20040805,1204047862746 for references: splitA: false,
splitB: true
2008-02-26 11:41:06,454 DEBUG org.apache.hadoop.hbase.HMaster:
HMaster.metaScanner regioninfo: {regionname:
pdc-docs,US20040151310_20040805,1204047862746, startKey:
<US20040151310_20040805>, endKey: <US20040157377_20040812>, encodedName:
516792513, offline: true, split: true, tableDesc: {name: pdc-docs, families:
{contents:={name: contents, max versions: 3, compression: NONE, in memory:
false, max length: 2147483647, bloom filter: none}, language:={name: language,
max versions: 3, compression: NONE, in memory: false, max length: 2147483647,
bloom filter: none}, mimetype:={name: mimetype, max versions: 3, compression:
NONE, in memory: false, max length: 2147483647, bloom filter: none}}}}, server:
192.168.105.29:60020, startCode: 1203949130420
2008-02-26 11:41:06,637 DEBUG org.apache.hadoop.hbase.HMaster:
pdc-docs,US20040154199_20040812,1204054791249 no longer has references to
pdc-docs,US20040151310_20040805,1204047862746
2008-02-26 11:41:06,637 INFO org.apache.hadoop.hbase.HMaster: Deleting region
pdc-docs,US20040151310_20040805,1204047862746 because daughter splits no longer
hold references
>From the regionserver (192.168.105.29):
2008-02-26 11:14:27,798 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache
flush for region pdc-docs,US20040151310_20040805,1204047862746. Size 2.9m
2008-02-26 11:14:28,197 DEBUG org.apache.hadoop.hbase.HRegion: Finished
memcache flush for region pdc-docs,US20040151310_20040805,1204047862746 in
399ms, sequenceid=1196699
2008-02-26 11:39:37,046 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache
flush for region pdc-docs,US20040151310_20040805,1204047862746. Size 7.3m
2008-02-26 11:39:37,605 DEBUG org.apache.hadoop.hbase.HRegion: Finished
memcache flush for region pdc-docs,US20040151310_20040805,1204047862746 in
559ms, sequenceid=1198452
2008-02-26 11:39:37,605 INFO org.apache.hadoop.hbase.HRegion: starting
compaction on region pdc-docs,US20040151310_20040805,1204047862746
2008-02-26 11:39:51,035 INFO org.apache.hadoop.hbase.HRegion: compaction
completed on region pdc-docs,US20040151310_20040805,1204047862746. Took 13sec
2008-02-26 11:39:51,246 INFO org.apache.hadoop.hbase.HRegion: Splitting
pdc-docs,US20040151310_20040805,1204047862746 because largest aggregate size is
256.7m and desired size is 256.0m
2008-02-26 11:39:51,249 DEBUG org.apache.hadoop.hbase.HRegion: compactions and
cache flushes disabled for region pdc-docs,US20040151310_20040805,1204047862746
2008-02-26 11:39:51,250 DEBUG org.apache.hadoop.hbase.HRegion: new updates and
scanners for region pdc-docs,US20040151310_20040805,1204047862746 disabled
2008-02-26 11:39:51,250 DEBUG org.apache.hadoop.hbase.HRegion: no more active
scanners for region pdc-docs,US20040151310_20040805,1204047862746
2008-02-26 11:39:51,250 DEBUG org.apache.hadoop.hbase.HRegion: no more row
locks outstanding on region pdc-docs,US20040151310_20040805,1204047862746
2008-02-26 11:39:51,250 DEBUG org.apache.hadoop.hbase.HRegionServer:
pdc-docs,US20040151310_20040805,1204047862746 closing (Adding to
retiringRegions)
2008-02-26 11:39:51,250 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache
flush for region pdc-docs,US20040151310_20040805,1204047862746. Size 18.5k
2008-02-26 11:39:51,442 DEBUG org.apache.hadoop.hbase.HRegion: Finished
memcache flush for region pdc-docs,US20040151310_20040805,1204047862746 in
192ms, sequenceid=1198501
2008-02-26 11:39:51,442 DEBUG org.apache.hadoop.hbase.HRegionServer:
pdc-docs,US20040151310_20040805,1204047862746 closed
2008-02-26 11:39:51,442 INFO org.apache.hadoop.hbase.HRegion: closed
pdc-docs,US20040151310_20040805,1204047862746
2008-02-26 11:39:51,442 DEBUG org.apache.hadoop.hbase.HRegionServer:
pdc-docs,US20040151310_20040805,1204047862746 closed
2008-02-26 11:39:51,873 INFO org.apache.hadoop.hbase.HRegionServer: region
split, META updated, and report to master all successful. Old
region=pdc-docs,US20040151310_20040805,1204047862746, new regions:
pdc-docs,US20040151310_20040805,1204054791249,
pdc-docs,US20040154199_20040812,1204054791249. Split took 0sec
Does that help?
> IllegalStateException thrown in client after region was split and deleted
> -------------------------------------------------------------------------
>
> Key: HBASE-471
> URL: https://issues.apache.org/jira/browse/HBASE-471
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.1.0
> Environment: Linux Debian, HBase 0.16.0
> Reporter: Lars George
>
> For some reason a client sometimes fails to locate a row with a
> IllegalStateException when the region was split and deleted.
> > [2008-02-25 16:12:39,171] ERROR [http-80-Processor20]
> > archive.MultilingualArchive - getDocument: An error occurred.
> > java.lang.IllegalStateException: region offline:
> > pdc-docs,US7039976_20060509,1203981958556
> > at
> > org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:432)
> > at
> > org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:343)
> > at
> > org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:306)
> > at org.apache.hadoop.hbase.HTable.getRegionLocation(HTable.java:102)
> > at org.apache.hadoop.hbase.HTable.get(HTable.java:280)
> Tracing the region on the master shows this:
> > 2008-02-25 16:09:38,761 DEBUG org.apache.hadoop.hbase.HMaster: Received
> > MSG_REGION_SPLIT :
> +pdc-docs,US7039976_20060509,1203981958556 from 192.168.105.21:60020
> > 2008-02-25 16:09:38,761 INFO org.apache.hadoop.hbase.HMaster: region
> > pdc-docs,US7039976_20060509,1203981958556 split. New regions
> +are: pdc-docs,US7039976_20060509,1203984578345,
> pdc-docs,US7046359_20060516,1203984578345
> > 2008-02-25 16:10:02,470 DEBUG org.apache.hadoop.hbase.HMaster:
> > HMaster.metaScanner regioninfo: {regionname:
> +pdc-docs,US7039976_20060509,1203981958556, startKey: <US7039976_20060509>,
> endKey: <US7053021_20060530>, encodedName: 1260314009,
> +offline: true, split: true, tableDesc: {name: pdc-docs, families:
> {contents:={name: contents, max versions: 3, compression: NONE, in
> +memory: false, max length: 2147483647, bloom filter: none}, language:={name:
> language, max versions: 3, compression: NONE, in
> +memory: false, max length: 2147483647, bloom filter: none}, mimetype:={name:
> mimetype, max versions: 3, compression: NONE, in
> +memory: false, max length: 2147483647, bloom filter: none}}}}, server:
> 192.168.105.21:60020, startCode: 1203949130468
> > 2008-02-25 16:10:02,513 DEBUG org.apache.hadoop.hbase.HMaster:
> > pdc-docs,US7039976_20060509,1203984578345 no longer has references
> +to pdc-docs,US7039976_20060509,1203981958556
> > 2008-02-25 16:10:02,516 DEBUG org.apache.hadoop.hbase.HMaster:
> > pdc-docs,US7046359_20060516,1203984578345 no longer has references
> +to pdc-docs,US7039976_20060509,1203981958556
> > 2008-02-25 16:10:02,516 INFO org.apache.hadoop.hbase.HMaster: Deleting
> > region pdc-docs,US7039976_20060509,1203981958556 because
> +daughter splits no longer hold references
> After discussion with st^ack it seems that the server is simply not retrying
> IllegalStateException but IOExceptions only. Also see HBASE-452 which should
> be addressed at the same time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
