[
https://issues.apache.org/jira/browse/HBASE-24515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Toshihiro Suzuki updated HBASE-24515:
-------------------------------------
Description:
When a client hits RPC timeout and sends a second RPC request for batch
Increment/Append but the first RPC is already processed actually, the nonce of
the RPC is saved in the RS.
In this case, for the second RPC, the RS just reads the previous result and
returns it to the client to avoid the Increment/Append is processed twice.
At that time, for batch Increment/Append, we try to create a Get object from a
CellScanner object in the following code:
[https://github.com/apache/hbase/blob/66452afc09d8b82927e5e58565f97939faa22c7b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L773-L776]
However, the CellScanner object is already consumed to create the
Increment/Append object in the following code, and it fails with the following
exception:
[https://github.com/apache/hbase/blob/66452afc09d8b82927e5e58565f97939faa22c7b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L757]
{code:java}
2020-06-06 14:09:06,153 WARN [hconnection-0x79c3903e-shared-pool3-t209]
client.AsyncRequestFutureImpl: id=1, table=REF_Test, attempt=3/36,
failureCount=1ops, last exception=org.apache.hadoop.hbase.DoNotRetryIOException:
org.apache.hadoop.hbase.DoNotRetryIOException: Cell count of 1 but at index 0
no cell returned: row: "xxxxxxxxxxx" mutate_type: INCREMENT timestamp:
9223372036854775807 durability: USE_DEFAULT time_range { from: 0 to:
9223372036854775807 } associated_cell_count: 1 nonce: 5281583076322914765
at
org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toGet(ProtobufUtil.java:934)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.increment(RSRpcServices.java:737)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:877)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2702)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42202)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
{code}
was:
When a client hits RPC timeout and sends a second RPC request for batch
Increment/Append but the first RPC is already processed actually, the nonce of
the RPC is saved in the RS.
In this case, for the second RPC, the RS just reads the previous result and
returns it to the client to avoid the Increment/Append is processed twice.
At that time, for batch Increment/Append, we try to create a Get object from a
CellScanner object in the following code:
[https://github.com/apache/hbase/blob/66452afc09d8b82927e5e58565f97939faa22c7b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L773-L776]
However, the CellScanner object is already consumed to create the
Increment/Append object as follows, and it fails:
[https://github.com/apache/hbase/blob/66452afc09d8b82927e5e58565f97939faa22c7b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L757]
> batch Increment/Append fails when retrying the RPC
> --------------------------------------------------
>
> Key: HBASE-24515
> URL: https://issues.apache.org/jira/browse/HBASE-24515
> Project: HBase
> Issue Type: Bug
> Reporter: Toshihiro Suzuki
> Assignee: Toshihiro Suzuki
> Priority: Major
>
> When a client hits RPC timeout and sends a second RPC request for batch
> Increment/Append but the first RPC is already processed actually, the nonce
> of the RPC is saved in the RS.
> In this case, for the second RPC, the RS just reads the previous result and
> returns it to the client to avoid the Increment/Append is processed twice.
> At that time, for batch Increment/Append, we try to create a Get object from
> a CellScanner object in the following code:
>
> [https://github.com/apache/hbase/blob/66452afc09d8b82927e5e58565f97939faa22c7b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L773-L776]
> However, the CellScanner object is already consumed to create the
> Increment/Append object in the following code, and it fails with the
> following exception:
>
> [https://github.com/apache/hbase/blob/66452afc09d8b82927e5e58565f97939faa22c7b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L757]
> {code:java}
> 2020-06-06 14:09:06,153 WARN [hconnection-0x79c3903e-shared-pool3-t209]
> client.AsyncRequestFutureImpl: id=1, table=REF_Test, attempt=3/36,
> failureCount=1ops, last
> exception=org.apache.hadoop.hbase.DoNotRetryIOException:
> org.apache.hadoop.hbase.DoNotRetryIOException: Cell count of 1 but at index 0
> no cell returned: row: "xxxxxxxxxxx" mutate_type: INCREMENT timestamp:
> 9223372036854775807 durability: USE_DEFAULT time_range { from: 0 to:
> 9223372036854775807 } associated_cell_count: 1 nonce: 5281583076322914765
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toGet(ProtobufUtil.java:934)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.increment(RSRpcServices.java:737)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:877)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2702)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42202)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
