[
https://issues.apache.org/jira/browse/PHOENIX-8053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Viraj Jasani updated PHOENIX-8053:
----------------------------------
Description:
Concurrent conditional UPSERT on the same row can fail, return uncommitted
data, or lose updates.
While one update on the data table is writing to its indexes, the update
releases the row lock. A later concurrent update on the same row builds on the
earlier update's pending state in memory (not from disk). However, this
tracking of the pending updates can cause issues.
Some of the issues as of today:
* *Non-retryable error after a no-op:* A conditional upsert (ON DUPLICATE KEY)
whose condition is false changes nothing and needs to return early, but it
stays recorded as the row's latest pending update. The next concurrent UPSERT
on the row finds it as completed and fails with a non-retryable
{*}IllegalArgumentException: the phase of the last batch cannot be POST{*}.
* *No-op result based on uncommitted data:* A conditional upsert whose
condition is false returns to the client right away. Its result, including the
returned row, can be based on the earlier update's pending state in memory. If
that update fails, the client was provided result based on uncommitted data.
* *Lost update after a failed update:* The row's latest pending update fails
while an earlier update is still pending. The next update reads the row from
disk (latest update failed) and misses the earlier pending write (which has not
completed yet), and then overwrites it once both commit.
* *Pending tracking removed too early:* Some updates never register as pending
on the row: whole-row upserts on an immutable table with a global or uncovered
index, or on a mutable table whose only indexes are uncovered. When such an
update completes, it still removes the row's pending entry, even the one that a
later update created. The next update reads from disk and misses a write that
is still in progress.
Also, the batches chained on a busy row (the pending ones and the ones already
done) stay in memory as long as the row has updates in flight.
Sample example for first case:
Caused by: org.apache.phoenix.execute.CommitException:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
action: org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException:
Failed to build index for unexpected reason!
at
org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:218)
at
org.apache.phoenix.hbase.index.IndexRegionObserver.preBatchMutate(IndexRegionObserver.java:706)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1002)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:999)
at
org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:558)
at
org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:631)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:999)
at
org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.prepareMiniBatchOperations(HRegion.java:3859)
at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4577)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4515)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4437)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1036)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:954)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:919)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2897)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45039)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:416)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Caused by: java.lang.IllegalArgumentException: the phase of the last batch
cannot be POST
at
org.apache.phoenix.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)
at
org.apache.phoenix.hbase.index.IndexRegionObserver.getCurrentRowStates(IndexRegionObserver.java:1264)
at
org.apache.phoenix.hbase.index.IndexRegionObserver.preBatchMutateWithExceptions(IndexRegionObserver.java:2029)
at
org.apache.phoenix.hbase.index.IndexRegionObserver.preBatchMutate(IndexRegionObserver.java:703)
... 18 more
was:
Concurrent conditional UPSERT on the same row can fail, return uncommitted
data, or lose updates.
While one update on the data table is writing to its indexes, the update
releases the row lock. A later concurrent update on the same row builds on the
earlier update's pending state in memory (not from disk). However, this
tracking of the pending updates can cause issues.
Some of the issues as of today:
* *Non-retryable error after a no-op:* A conditional upsert (ON DUPLICATE KEY)
whose condition is false changes nothing and needs to return early, but it
stays recorded as the row's latest pending update. The next concurrent UPSERT
on the row finds it as completed and fails with a non-retryable
{*}IllegalArgumentException: the phase of the last batch cannot be POST{*}.
* *No-op result based on uncommitted data:* A conditional upsert whose
condition is false returns to the client right away. Its result, including the
returned row, can be based on the earlier update's pending state in memory. If
that update fails, the client was provided result based on uncommitted data.
* *Lost update after a failed update:* The row's latest pending update fails
while an earlier update is still pending. The next update reads the row from
disk (latest update failed) and misses the earlier pending write (which has not
completed yet), and then overwrites it once both commit.
* *Pending tracking removed too early:* Some updates never register as pending
on the row: whole-row upserts on an immutable table with a global or uncovered
index, or on a mutable table whose only indexes are uncovered. When such an
update completes, it still removes the row's pending entry, even the one that a
later update created. The next update reads from disk and misses a write that
is still in progress.
Also, the batches chained on a busy row (the pending ones and the ones already
done) stay in memory as long as the row has updates in flight.
Sample Exception for first case:
{code:java}
Caused by: org.apache.phoenix.execute.CommitException:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
action: org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException:
Failed to build index for unexpected reason! at
org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:218)
at
org.apache.phoenix.hbase.index.IndexRegionObserver.preBatchMutate(IndexRegionObserver.java:706)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1002)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:999)
at
org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:558)
at
org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:631)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:999)
at
org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.prepareMiniBatchOperations(HRegion.java:3859)
at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4577)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4515) at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4437) at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1036)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:954)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:919)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2897)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45039)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:416) at
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) at
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102) at
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)Caused by:
java.lang.IllegalArgumentException: the phase of the last batch cannot be POST
at
org.apache.phoenix.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)
at
org.apache.phoenix.hbase.index.IndexRegionObserver.getCurrentRowStates(IndexRegionObserver.java:1264)
at
org.apache.phoenix.hbase.index.IndexRegionObserver.preBatchMutateWithExceptions(IndexRegionObserver.java:2029)
at
org.apache.phoenix.hbase.index.IndexRegionObserver.preBatchMutate(IndexRegionObserver.java:703)
... 18 more
{code}
> Concurrency bugs related to conditional UPSERTs on same row
> -----------------------------------------------------------
>
> Key: PHOENIX-8053
> URL: https://issues.apache.org/jira/browse/PHOENIX-8053
> Project: Phoenix
> Issue Type: Bug
> Reporter: Viraj Jasani
> Assignee: Viraj Jasani
> Priority: Major
>
> Concurrent conditional UPSERT on the same row can fail, return uncommitted
> data, or lose updates.
> While one update on the data table is writing to its indexes, the update
> releases the row lock. A later concurrent update on the same row builds on
> the earlier update's pending state in memory (not from disk). However, this
> tracking of the pending updates can cause issues.
> Some of the issues as of today:
> * *Non-retryable error after a no-op:* A conditional upsert (ON DUPLICATE
> KEY) whose condition is false changes nothing and needs to return early, but
> it stays recorded as the row's latest pending update. The next concurrent
> UPSERT on the row finds it as completed and fails with a non-retryable
> {*}IllegalArgumentException: the phase of the last batch cannot be POST{*}.
> * *No-op result based on uncommitted data:* A conditional upsert whose
> condition is false returns to the client right away. Its result, including
> the returned row, can be based on the earlier update's pending state in
> memory. If that update fails, the client was provided result based on
> uncommitted data.
> * *Lost update after a failed update:* The row's latest pending update fails
> while an earlier update is still pending. The next update reads the row from
> disk (latest update failed) and misses the earlier pending write (which has
> not completed yet), and then overwrites it once both commit.
> * *Pending tracking removed too early:* Some updates never register as
> pending on the row: whole-row upserts on an immutable table with a global or
> uncovered index, or on a mutable table whose only indexes are uncovered. When
> such an update completes, it still removes the row's pending entry, even the
> one that a later update created. The next update reads from disk and misses a
> write that is still in progress.
> Also, the batches chained on a busy row (the pending ones and the ones
> already done) stay in memory as long as the row has updates in flight.
> Sample example for first case:
> Caused by: org.apache.phoenix.execute.CommitException:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException:
> Failed to build index for unexpected reason!
> at
> org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:218)
> at
> org.apache.phoenix.hbase.index.IndexRegionObserver.preBatchMutate(IndexRegionObserver.java:706)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1002)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:999)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:558)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:631)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:999)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.prepareMiniBatchOperations(HRegion.java:3859)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4577)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4515)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4437)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1036)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:954)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:919)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2897)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45039)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:416)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
> Caused by: java.lang.IllegalArgumentException: the phase of the last batch
> cannot be POST
> at
> org.apache.phoenix.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)
> at
> org.apache.phoenix.hbase.index.IndexRegionObserver.getCurrentRowStates(IndexRegionObserver.java:1264)
> at
> org.apache.phoenix.hbase.index.IndexRegionObserver.preBatchMutateWithExceptions(IndexRegionObserver.java:2029)
> at
> org.apache.phoenix.hbase.index.IndexRegionObserver.preBatchMutate(IndexRegionObserver.java:703)
> ... 18 more
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
