Hi
We are having a issue while running the job as it hangs forever at 99%
(reducer) inserting into the table , hbase hbck -details finds
inconsistency in the table I am trying to insert data as :--
Chain of regions in table NAM_CLUSTERKEYS3 is broken; edges does not
contain
00000000|00000000000000000USA|0000240b|000000000000000LAKE,|002ff027|000
0000000CONSULTING|006e8f90|000000000000000000NY:0
I tried hbase hbck -fix but does not help.
Tried scanning the '.META' table
scan
'.META.',{STARTROW=>'NAM_CLUSTERKEYS3,00000000|00000000000000000USA|0000
240b|000000000000000LAKE',LIMIT=>1}
Getting this error:--
ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying
to contact region server doop6. for region .META.,,1, row
'NAM_CLUSTERKEYS3,00000000|00000000000000000USA|0000240b|000000000000000
LAKE', but failed after 7 attempts.
Exceptions:
java.io.IOException: java.io.IOException:
java.lang.IllegalArgumentException: No 44 in
<WKNAM_CLUSTERKEYS3,00000000|00000000000000000USA|0000240b|0000000000000
00LAK????????>, length=58, offset=27
at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE
(HRegionServer.java:992)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE
(HRegionServer.java:981)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionSe
rver.java:1783)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Caused by: java.lang.IllegalArgumentException: No 44 in
<WKNAM_CLUSTERKEYS3,00000000|00000000000000000USA|0000240b|0000000000000
00LAK????????>, length=58, offset=27
at
org.apache.hadoop.hbase.KeyValue.getRequiredDelimiterInReverse(KeyValue.
java:1281)
at
org.apache.hadoop.hbase.KeyValue$MetaKeyComparator.compareRows(KeyValue.
java:1827)
at
org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:186
6)
at
org.apache.hadoop.hbase.util.Bytes.binarySearch(Bytes.java:1159)
at
org.apache.hadoop.hbase.io.hfile.HFile$BlockIndex.blockContainingKey(HFi
le.java:1618)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.blockContainingKey(HFile.j
ava:918)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:
1296)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(Stor
eFileScanner.java:136)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScan
ner.java:96)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.ja
va:77)
at
org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1341)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.<init>(HRegio
n.java:2269)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateInternalScanner(
HRegion.java:1126)
at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:111
8)
at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:110
2)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionSe
rver.java:1781)
... 5 more
java.io.IOException: java.io.IOException:
java.lang.IllegalArgumentException: No 44 in
<WKNAM_CLUSTERKEYS3,00000000|00000000000000000USA|0000240b|0000000000000
00LAK????????>, length=58, offset=27
at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE
(HRegionServer.java:992)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE
(HRegionServer.java:981)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionSe
rver.java:1783)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Caused by: java.lang.IllegalArgumentException: No 44 in
<WKNAM_CLUSTERKEYS3,00000000|00000000000000000USA|0000240b|0000000000000
00LAK????????>, length=58, offset=27
at
org.apache.hadoop.hbase.KeyValue.getRequiredDelimiterInReverse(KeyValue.
java:1281)
at
org.apache.hadoop.hbase.KeyValue$MetaKeyComparator.compareRows(KeyValue.
java:1827)
at
org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:186
6)
at
org.apache.hadoop.hbase.util.Bytes.binarySearch(Bytes.java:1159)
at
org.apache.hadoop.hbase.io.hfile.HFile$BlockIndex.blockContainingKey(HFi
le.java:1618)
at org.apache.hadoop.hbase.io.hfile.HFile$Reader.blockContaini
How can I fix the problem in the table . Any help would be appreciated.
