Viraj Jasani created PHOENIX-8053:
-------------------------------------
Summary: Concurrency bugs related to conditional UPSERTs on same
row
Key: PHOENIX-8053
URL: https://issues.apache.org/jira/browse/PHOENIX-8053
Project: Phoenix
Issue Type: Bug
Reporter: Viraj Jasani
Concurrent conditional UPSERT on the same row can fail, return uncommitted
data, or lose updates.
While one update on the data table is writing to its indexes, the update
releases the row lock. A later concurrent update on the same row builds on the
earlier update's pending state in memory (not from disk). However, this
tracking of the pending updates can cause issues.
Some of the issues as of today:
* *Non-retryable error after a no-op:* A conditional upsert (ON DUPLICATE KEY)
whose condition is false changes nothing and needs to return early, but it
stays recorded as the row's latest pending update. The next concurrent UPSERT
on the row finds it as completed and fails with a non-retryable
{*}IllegalArgumentException: the phase of the last batch cannot be POST{*}.
* *No-op result based on uncommitted data:* A conditional upsert whose
condition is false returns to the client right away. Its result, including the
returned row, can be based on the earlier update's pending state in memory. If
that update fails, the client was provided result based on uncommitted data.
* *Lost update after a failed update:* The row's latest pending update fails
while an earlier update is still pending. The next update reads the row from
disk (latest update failed) and misses the earlier pending write (which has not
completed yet), and then overwrites it once both commit.
* *Pending tracking removed too early:* Some updates never register as pending
on the row: whole-row upserts on an immutable table with a global or uncovered
index, or on a mutable table whose only indexes are uncovered. When such an
update completes, it still removes the row's pending entry, even the one that a
later update created. The next update reads from disk and misses a write that
is still in progress.
Also, the batches chained on a busy row (the pending ones and the ones already
done) stay in memory as long as the row has updates in flight.
Sample Exception for first case:
{code:java}
Caused by: org.apache.phoenix.execute.CommitException:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
action: org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException:
Failed to build index for unexpected reason! at
org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:218)
at
org.apache.phoenix.hbase.index.IndexRegionObserver.preBatchMutate(IndexRegionObserver.java:706)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1002)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:999)
at
org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:558)
at
org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:631)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:999)
at
org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.prepareMiniBatchOperations(HRegion.java:3859)
at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4577)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4515) at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4437) at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1036)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:954)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:919)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2897)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45039)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:416) at
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) at
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102) at
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)Caused by:
java.lang.IllegalArgumentException: the phase of the last batch cannot be POST
at
org.apache.phoenix.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)
at
org.apache.phoenix.hbase.index.IndexRegionObserver.getCurrentRowStates(IndexRegionObserver.java:1264)
at
org.apache.phoenix.hbase.index.IndexRegionObserver.preBatchMutateWithExceptions(IndexRegionObserver.java:2029)
at
org.apache.phoenix.hbase.index.IndexRegionObserver.preBatchMutate(IndexRegionObserver.java:703)
... 18 more
{code}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
