[
https://issues.apache.org/jira/browse/HBASE-29678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Roudnitsky updated HBASE-29678:
--------------------------------------
Description:
If a scan with a filter using BigDecimalComparator encounters a row/column/byte
array which was not encoded properly using Bytes.toBytes(BigDecimal) and is too
short to be a BigDecimal, the scan RPC will fail with a NullPointerException,
and the client will retry excessively until exhausting all retries, and the
operation will fail with RetriesExhausted and a trace that looks like:
{code:java}
RetriesExhausted Failed after attempts=13, exceptions:
2025-10-20T19:31:23.525Z, java.io.IOException: java.io.IOException: Cannot read
field "scale" because "val" is null
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:525)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Caused by: java.lang.NullPointerException: Cannot read field "scale" because
"val" is null
at java.base/java.math.BigDecimal.compareTo(BigDecimal.java:3124)
at
org.apache.hadoop.hbase.filter.BigDecimalComparator.compareTo(BigDecimalComparator.java:72)
at
org.apache.hadoop.hbase.PrivateCellUtil.compareValue(PrivateCellUtil.java:1341)
at
org.apache.hadoop.hbase.filter.ColumnValueFilter.compareValue(ColumnValueFilter.java:128)
at
org.apache.hadoop.hbase.filter.ColumnValueFilter.filterCell(ColumnValueFilter.java:111)
at
org.apache.hadoop.hbase.regionserver.querymatcher.UserScanQueryMatcher.matchColumn(UserScanQueryMatcher.java:166)
at
org.apache.hadoop.hbase.regionserver.querymatcher.NormalUserScanQueryMatcher.match(NormalUserScanQueryMatcher.java:80)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:653)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:153)
at
org.apache.hadoop.hbase.regionserver.RegionScannerImpl.populateResult(RegionScannerImpl.java:348)
at
org.apache.hadoop.hbase.regionserver.RegionScannerImpl.nextInternal(RegionScannerImpl.java:520)
at
org.apache.hadoop.hbase.regionserver.RegionScannerImpl.nextRaw(RegionScannerImpl.java:285)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3430)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3703)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45423)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:458)
... 3 more
2025-10-20T19:31:23.645Z, java.io.IOException: java.io.IOException: Cannot read
field "scale" because "val" is null
... {code}
+Root cause+
BigDecimalComparator#compareTo relies on ByteBufferUtils.toBigDecimal to
convert the input byte array to BigDecimal, and ByteBufferUtils.toBigDecimal
will return null if the byte array is too short. We don’t do a null check
before doing the comparison , which leads to a NullPointerException that
bubbles up the region server call stack which the client will treat as
retryable exception
+Proposed solution+
Add a null check before doing the comparison in BigDecimalComparator - if we
have null then the input byte array conversion to BigDecimal failed and we
should throw an exception that explains the cause clearly - input byte array is
too short to be a BigDecimal - and make sure the exception is handled as a
DoNotRetryIOException in the filter layer.
See parent issue HBASE-29672
was:
If a scan with a filter using BigDecimalComparator encounters a row/column/byte
array which was not encoded properly using Bytes.toBytes(BigDecimal) and is too
short to be a BigDecimal, the scan RPC will fail with a NullPointerException,
and the client will retry excessively until exhausting all retries, and the
operation will fail with RetriesExhausted and a trace that looks like:
{code:java}
RetriesExhausted Failed after attempts=13, exceptions:
2025-10-20T19:31:23.525Z, java.io.IOException: java.io.IOException: Cannot read
field "scale" because "val" is null
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:525)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Caused by: java.lang.NullPointerException: Cannot read field "scale" because
"val" is null
at java.base/java.math.BigDecimal.compareTo(BigDecimal.java:3124)
at
org.apache.hadoop.hbase.filter.BigDecimalComparator.compareTo(BigDecimalComparator.java:72)
at
org.apache.hadoop.hbase.PrivateCellUtil.compareValue(PrivateCellUtil.java:1341)
at
org.apache.hadoop.hbase.filter.ColumnValueFilter.compareValue(ColumnValueFilter.java:128)
at
org.apache.hadoop.hbase.filter.ColumnValueFilter.filterCell(ColumnValueFilter.java:111)
at
org.apache.hadoop.hbase.regionserver.querymatcher.UserScanQueryMatcher.matchColumn(UserScanQueryMatcher.java:166)
at
org.apache.hadoop.hbase.regionserver.querymatcher.NormalUserScanQueryMatcher.match(NormalUserScanQueryMatcher.java:80)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:653)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:153)
at
org.apache.hadoop.hbase.regionserver.RegionScannerImpl.populateResult(RegionScannerImpl.java:348)
at
org.apache.hadoop.hbase.regionserver.RegionScannerImpl.nextInternal(RegionScannerImpl.java:520)
at
org.apache.hadoop.hbase.regionserver.RegionScannerImpl.nextRaw(RegionScannerImpl.java:285)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3430)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3703)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45423)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:458)
... 3 more
2025-10-20T19:31:23.645Z, java.io.IOException: java.io.IOException: Cannot read
field "scale" because "val" is null
... {code}
+Root cause+
BigDecimalComparator#compareTo relies on ByteBufferUtils.toBigDecimal to
convert the input byte array to BigDecimal, and ByteBufferUtils.toBigDecimal
will return null if the byte array is too short. We don’t do a null check
before doing the comparison , which leads to a NullPointerException that
bubbles up the region server call stack which the client will treat as
retryable exception
+Proposed solution+
Add a null check before doing the comparison in BigDecimalComparator - if we
have null then the input byte array conversion to BigDecimal failed and we
should throw an exception that explains the cause clearly - input byte array is
too short to be a BigDecimal - and make sure the exception is handled as a
DoNotRetryIOException in the filter layer.
See parent issue
> Filtering with BigDecimalComparator can fail non-gracefully with
> NullPointerException
> -------------------------------------------------------------------------------------
>
> Key: HBASE-29678
> URL: https://issues.apache.org/jira/browse/HBASE-29678
> Project: HBase
> Issue Type: Bug
> Components: Filters
> Reporter: Daniel Roudnitsky
> Assignee: Daniel Roudnitsky
> Priority: Major
>
> If a scan with a filter using BigDecimalComparator encounters a
> row/column/byte array which was not encoded properly using
> Bytes.toBytes(BigDecimal) and is too short to be a BigDecimal, the scan RPC
> will fail with a NullPointerException, and the client will retry excessively
> until exhausting all retries, and the operation will fail with
> RetriesExhausted and a trace that looks like:
> {code:java}
> RetriesExhausted Failed after attempts=13, exceptions:
> 2025-10-20T19:31:23.525Z, java.io.IOException: java.io.IOException: Cannot
> read field "scale" because "val" is null
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:525)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
> Caused by: java.lang.NullPointerException: Cannot read field "scale" because
> "val" is null
> at java.base/java.math.BigDecimal.compareTo(BigDecimal.java:3124)
> at
> org.apache.hadoop.hbase.filter.BigDecimalComparator.compareTo(BigDecimalComparator.java:72)
> at
> org.apache.hadoop.hbase.PrivateCellUtil.compareValue(PrivateCellUtil.java:1341)
> at
> org.apache.hadoop.hbase.filter.ColumnValueFilter.compareValue(ColumnValueFilter.java:128)
> at
> org.apache.hadoop.hbase.filter.ColumnValueFilter.filterCell(ColumnValueFilter.java:111)
> at
> org.apache.hadoop.hbase.regionserver.querymatcher.UserScanQueryMatcher.matchColumn(UserScanQueryMatcher.java:166)
> at
> org.apache.hadoop.hbase.regionserver.querymatcher.NormalUserScanQueryMatcher.match(NormalUserScanQueryMatcher.java:80)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:653)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:153)
> at
> org.apache.hadoop.hbase.regionserver.RegionScannerImpl.populateResult(RegionScannerImpl.java:348)
> at
> org.apache.hadoop.hbase.regionserver.RegionScannerImpl.nextInternal(RegionScannerImpl.java:520)
> at
> org.apache.hadoop.hbase.regionserver.RegionScannerImpl.nextRaw(RegionScannerImpl.java:285)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3430)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3703)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45423)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:458)
> ... 3 more
> 2025-10-20T19:31:23.645Z, java.io.IOException: java.io.IOException: Cannot
> read field "scale" because "val" is null
> ... {code}
> +Root cause+
> BigDecimalComparator#compareTo relies on ByteBufferUtils.toBigDecimal to
> convert the input byte array to BigDecimal, and ByteBufferUtils.toBigDecimal
> will return null if the byte array is too short. We don’t do a null check
> before doing the comparison , which leads to a NullPointerException that
> bubbles up the region server call stack which the client will treat as
> retryable exception
> +Proposed solution+
> Add a null check before doing the comparison in BigDecimalComparator - if we
> have null then the input byte array conversion to BigDecimal failed and we
> should throw an exception that explains the cause clearly - input byte array
> is too short to be a BigDecimal - and make sure the exception is handled as a
> DoNotRetryIOException in the filter layer.
> See parent issue HBASE-29672
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
