[
https://issues.apache.org/jira/browse/HBASE-29654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18030974#comment-18030974
]
Daniel Roudnitsky commented on HBASE-29654:
-------------------------------------------
I have split this up into three smaller PRs/work items:
# HBASE-29675 - Add a more descriptive OffsetOutOfBoundsException to
BinaryComponentComparator which subclasses ArrayIndexOutOfBoundsIndex -
provides a much clearer exception message without changing any other behavior -
simple change without dependency on anything else
# HBASE-29672 - For all filters which take a comparator, wrap runtime
exceptions from the comparator layer in HBaseIOException with a clear message
for the client which will be retried, that PR is also ready/independent of
HBASE-29675
# HBASE-29676 - Final piece needed to handle OffsetOutOfBoundsException as
DoNotRetryIOException - will be a very small PR once 1 & 2 are merged for this
special case
Thank you very much [~zhangduo] for you feedback/review
> Filtering with BinaryComponentComparator can fail non-gracefully with
> ArrayIndexOutOfBoundsException
> ----------------------------------------------------------------------------------------------------
>
> Key: HBASE-29654
> URL: https://issues.apache.org/jira/browse/HBASE-29654
> Project: HBase
> Issue Type: Bug
> Components: Filters
> Reporter: Daniel Roudnitsky
> Assignee: Daniel Roudnitsky
> Priority: Major
> Labels: pull-request-available
>
> +Summary+
> Filtering with
> [BinaryComponentComparator|https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/filter/BinaryComponentComparator.html]
> (HBASE-22969) enables a user to compare against a subset of a byte array by
> specifying an offset into the byte array from which the comparison should
> start. The issue here is that if the offset provided by the user is longer
> than a byte array encountered by the scan query (e.g an offset of 40 is used,
> and there is a "short" byte array of length 38 somewhere in the table), when
> the scan query reaches the "short" byte array the query will fail in a
> nongraceful manner - the scan RPC fails with a mysterious
> ArrayIndexOutOfBoundsException exception, the client continues to retry the
> failed RPC excessively until exhausting all retries, and the client operation
> ultimately fails out with an unintuitive RetriesExhaustedException with a
> trace that can look like:
> {code:java}
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=2, exceptions:
> 2025-09-26T19:09:35.531Z,
> RpcRetryingCaller{globalStartTime=2025-09-26T19:09:35.412Z, pause=1000,
> maxAttempts=2}, java.io.IOException: java.io.IOException
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:451)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:139)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349)
> Caused by: java.lang.ArrayIndexOutOfBoundsException
> 2025-09-26T19:09:36.659Z,
> RpcRetryingCaller{globalStartTime=2025-09-26T19:09:35.412Z, pause=1000,
> maxAttempts=2}, java.io.IOException: java.io.IOException
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:451)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:139)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349)
> Caused by: java.lang.ArrayIndexOutOfBoundsException
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:142)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:73)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at java.base/java.lang.Thread.run(Thread.java:840){code}
> +Root cause+
> Serverside the scan request handler will throw an unhandled/unexpected
> ArrayIndexOutOfBoundsException during request processing when applying the
> filter with BinaryComponentComparator and attempting to do an unchecked byte
> array comparison using an offset which extends beyond the length of the
> "short" byte array that is being read, when this happens the client will
> treat the remote exception from the server as an IOException (which wraps the
> remote exception), and because IOException is retryable, the client will
> proceed to exhaust all of its retries re-running the same RPC (with all
> retries guaranteed to fail).
> +Proposed solution+
> The semantics of BinaryComponentComparator have always been to fail when
> encountering unexpected/incompatible length data. The solution here should be
> to preserve the existing semantics to maintain compatibility, but the
> behavior should be enhanced so that when it fails it does so gracefully
> without excessive retries and with a clear error message provided to the
> user.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
