[
https://issues.apache.org/jira/browse/HBASE-22761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17494500#comment-17494500
]
Xiaolin Ha edited comment on HBASE-22761 at 2/18/22, 9:49 AM:
--------------------------------------------------------------
We have also met this problem after HBASE-22539 .
{code:java}
022-01-04 03:57:04,027 INFO [RS_LOG_REPLAY_OPS-1] wal.HdfsWALSplitter:
Processed 23 edits across 1 regions; edits skipped=0; log
file=hdfs://xr-data-hbase/hbase/WALs/xr-data-hbase230.mt,16020,1640632682455-splitting/xr-data-hbase230.mt%2C16020%2C1640632682455.xr-data-hbase230.mt%2C16020%2C1640632682455.regiongroup-0.1641239364415,
length=343709223, corrupted=false, progress failed=false
2022-01-04 03:57:04,027 INFO [RS_LOG_REPLAY_OPS-1] handler.WALSplitterHandler:
worker xr-data-hbase157.mt,16020,1640052851334 done with task
org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination$ZkSplitTaskDetails@d928e53
in 734ms
2022-01-04 03:57:04,027 ERROR [RS_LOG_REPLAY_OPS-1] executor.EventHandler:
Caught throwable while processing event RS_LOG_REPLAY
java.lang.ArrayIndexOutOfBoundsException: -27662
at org.apache.hadoop.hbase.KeyValue.getFamilyLength(KeyValue.java:1440)
at org.apache.hadoop.hbase.KeyValue.getFamilyLength(KeyValue.java:1433)
at org.apache.hadoop.hbase.CellUtil.matchingFamily(CellUtil.java:1211)
at org.apache.hadoop.hbase.CellUtil.matchingFamily(CellUtil.java:1201)
at
org.apache.hadoop.hbase.regionserver.wal.WALEdit.isMetaEditFamily(WALEdit.java:191)
at
org.apache.hadoop.hbase.regionserver.wal.WALEdit.isMetaEdit(WALEdit.java:200)
at
org.apache.hadoop.hbase.wal.HdfsWALSplitter.splitLogFile(HdfsWALSplitter.java:250)
at
org.apache.hadoop.hbase.wal.HdfsWALSplitter.splitLogFile(HdfsWALSplitter.java:118)
at
org.apache.hadoop.hbase.regionserver.HdfsSplitLogWorker$1.exec(HdfsSplitLogWorker.java:73)
at
org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:72)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) {code}
was (Author: xiaolin ha):
We have also met this problem after HBASE-22539 .
{code:java}
022-01-04 03:57:04,027 INFO [RS_LOG_REPLAY_OPS-1] wal.HdfsWALSplitter:
Processed 23 edits across 1 regions; edits skipped=0; log
file=hdfs://xr-data-hbase/hbase/WALs/xr-data-hbase230.mt,16020,1640632682455-splitting/xr-data-hbase230.mt%2C16020%2C1640632682455.xr-data-hbase230.mt%2C16020%2C1640632682455.regiongroup-0.1641239364415,
length=343709223, corrupted=false, progress failed=false
2022-01-04 03:57:04,027 INFO [RS_LOG_REPLAY_OPS-1] handler.WALSplitterHandler:
worker xr-data-hbase157.mt,16020,1640052851334 done with task
org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination$ZkSplitTaskDetails@d928e53
in 734ms
2022-01-04 03:57:04,027 ERROR [RS_LOG_REPLAY_OPS-1] executor.EventHandler:
Caught throwable while processing event RS_LOG_REPLAY
java.lang.ArrayIndexOutOfBoundsException: -27662
at org.apache.hadoop.hbase.KeyValue.getFamilyLength(KeyValue.java:1440)
at org.apache.hadoop.hbase.KeyValue.getFamilyLength(KeyValue.java:1433)
at org.apache.hadoop.hbase.CellUtil.matchingFamily(CellUtil.java:1211)
421033,18-25 43%
at org.apache.hadoop.hbase.CellUtil.matchingFamily(CellUtil.java:1201)
at
org.apache.hadoop.hbase.regionserver.wal.WALEdit.isMetaEditFamily(WALEdit.java:191)
421035,18-25 43%
at
org.apache.hadoop.hbase.regionserver.wal.WALEdit.isMetaEdit(WALEdit.java:200)
421036,18-25 43%
at
org.apache.hadoop.hbase.wal.HdfsWALSplitter.splitLogFile(HdfsWALSplitter.java:250)
at
org.apache.hadoop.hbase.wal.HdfsWALSplitter.splitLogFile(HdfsWALSplitter.java:118)
421038,18-25 43%
at
org.apache.hadoop.hbase.regionserver.HdfsSplitLogWorker$1.exec(HdfsSplitLogWorker.java:73)
at
org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:72)
421040,18-25 43%
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
421041,18-25 43%
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) {code}
> Caught ArrayIndexOutOfBoundsException while processing event RS_LOG_REPLAY
> --------------------------------------------------------------------------
>
> Key: HBASE-22761
> URL: https://issues.apache.org/jira/browse/HBASE-22761
> Project: HBase
> Issue Type: Bug
> Affects Versions: 2.1.1
> Reporter: casuallc
> Priority: Major
> Attachments: tmp
>
>
> RegionServer exists when error happen
> {code:java}
> 2019-07-29 20:51:09,726 INFO [RS_LOG_REPLAY_OPS-regionserver/h1:16020-0]
> wal.WALSplitter: Processed 0 edits across 0 regions; edits skipped=0; log
> file=hdfs://cluster1/hbase/WALs/h2,16020,1564216856546-splitting/h2%2C16020%2C1564216856546.1564398538121,
> length=615233, corrupted=false, progress failed=false
> 2019-07-29 20:51:09,726 INFO [RS_LOG_REPLAY_OPS-regionserver/h1:16020-0]
> handler.WALSplitterHandler: Worker h1,16020,1564404572589 done with task
> org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination$ZkSplitTaskDetails@577da0d3
> in 84892ms. Status = null
> 2019-07-29 20:51:09,726 ERROR [RS_LOG_REPLAY_OPS-regionserver/h1:16020-0]
> executor.EventHandler: Caught throwable while processing event RS_LOG_REPLAY
> java.lang.ArrayIndexOutOfBoundsException: 16403
> at org.apache.hadoop.hbase.KeyValue.getFamilyLength(KeyValue.java:1365)
> at org.apache.hadoop.hbase.KeyValue.getFamilyLength(KeyValue.java:1358)
> at
> org.apache.hadoop.hbase.PrivateCellUtil.matchingFamily(PrivateCellUtil.java:735)
> at org.apache.hadoop.hbase.CellUtil.matchingFamily(CellUtil.java:816)
> at org.apache.hadoop.hbase.wal.WALEdit.isMetaEditFamily(WALEdit.java:143)
> at org.apache.hadoop.hbase.wal.WALEdit.isMetaEdit(WALEdit.java:148)
> at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:297)
> at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:195)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:100)
> at
> org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:70)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2019-07-29 20:51:09,730 ERROR [RS_LOG_REPLAY_OPS-regionserver/h1:16020-0]
> regionserver.HRegionServer: ***** ABORTING region server
> h1,16020,1564404572589: Caught throwable while processing event RS_LOG_REPLAY
> *****
> java.lang.ArrayIndexOutOfBoundsException: 16403
> at org.apache.hadoop.hbase.KeyValue.getFamilyLength(KeyValue.java:1365)
> at org.apache.hadoop.hbase.KeyValue.getFamilyLength(KeyValue.java:1358)
> at
> org.apache.hadoop.hbase.PrivateCellUtil.matchingFamily(PrivateCellUtil.java:735)
> at org.apache.hadoop.hbase.CellUtil.matchingFamily(CellUtil.java:816)
> at org.apache.hadoop.hbase.wal.WALEdit.isMetaEditFamily(WALEdit.java:143)
> at org.apache.hadoop.hbase.wal.WALEdit.isMetaEdit(WALEdit.java:148)
> at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:297)
> at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:195)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:100)
> at
> org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:70)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
