shashank created CAMEL-25758:
--------------------------------
Summary: camel-azure-storage-blob - consumer with moveAfterRead
deletes the source blob although the copy to the destination container failed,
was aborted or its status could not be read: the blob is lost
Key: CAMEL-25758
URL: https://issues.apache.org/jira/browse/CAMEL-25758
Project: Camel
Issue Type: Bug
Components: camel-azure
Reporter: shashank
Assignee: shashank
With {{moveAfterRead=true}} the blob consumer moves a consumed blob in
{{BlobConsumer.processCommit}} (main e13303ef1ef7, lines 283-286) as copy +
delete:
{code:java}
copyBlobToDestination(containerName, blobName, destinationContainer,
destinationBlobName);
deleteBlob(containerName, blobName);
{code}
and {{copyBlobToDestination}} (lines 362-363) runs:
{code:java}
destBlobClient.beginCopy(sourceBlobClient.getBlobUrl(), Duration.ofSeconds(2))
.waitForCompletion(timeout);
{code}
The result of {{waitForCompletion}} is not looked at. A Copy Blob is an
asynchronous operation of the service: its status is {{pending}} until it ends
as {{success}}, {{aborted}} or {{failed}} (Get Blob Properties,
{{x-ms-copy-status}}). The {{SyncPoller}} of {{BlobClientBase.beginCopy}}
(azure-storage-blob 12.32, read from the jar) does not throw when the copy does
not succeed: it returns the last poll response once its status is complete,
that is {{FAILED}} for a failed copy, {{USER_CANCELLED}} for an aborted copy,
and the custom complete status {{POLLING_FAILED}} when reading the status of a
pending copy failed (an exception of {{getProperties()}} is caught in
{{onPoll}}). Apart from a copy request that the service refuses
({{BlobStorageException}} when the copy starts), only a timeout throws
({{RuntimeException}} with a {{TimeoutException}}). In all three cases the
consumer deletes the source blob next. The destination of a failed or aborted
copy has no content (Copy Blob remarks: an aborted copy leaves a blob of 0
bytes, a copy that times out on the service leaves an empty blob), and with
{{POLLING_FAILED}} the copy can still be pending when the source is deleted
("any changes to the source while the copy operation is in progress will cause
that operation to fail", Copy Blob remarks). The blob is gone from both
containers and Camel logs nothing.
How likely: the move is always within one storage account (one
{{BlobServiceClient}}), and Microsoft Learn says such a copy "can complete
synchronously". A copy that completes synchronously is read back as {{success}}
and is not affected. The loss needs a copy that the service runs asynchronously
and that then fails or is aborted, or a failed status read while it is pending.
How often same-account copies are asynchronous is not documented.
The copy was a synchronous {{copyFromUrl}} up to camel-4.18.x (it throws on
failure); CAMEL-24158 switched to {{beginCopy}} for the authorization of the
copy source.
*Fix* (one class): {{processCommit}} deletes the source blob only when the copy
ended with {{SUCCESSFULLY_COMPLETED}}. Otherwise it reports the copy status to
the consumer's exception handler and keeps the blob, which is consumed again on
the next poll, the same as after a copy request that fails with a
{{BlobStorageException}} today. A copy that keeps failing is therefore retried
at every poll with one exception handler call each time, as a copy that is
refused already is. A copy reported as {{POLLING_FAILED}} that had in fact
succeeded is copied and routed again on the next poll (at least once instead of
a possible loss); if the status can never be read (credentials that may write
but not read the destination container), every move is reported and retried
this way, where main deleted the source without knowing the outcome. A copy
that times out is unchanged: the exception leaves the on completion, Camel logs
it and the source blob is kept.
*Tests*: new {{BlobConsumerMoveAfterReadCopyStatusTest}} (4 tests, Mockito
clients as in {{BlobConsumerInProgressTest}}; the poller returns the given
status): the source blob is kept and the exception handler is called when the
copy ends {{FAILED}}, {{USER_CANCELLED}} or {{POLLING_FAILED}}; the source blob
is deleted after a successful copy. The three "kept" tests fail on main
({{NeverWantedButInvoked: blobClient.delete()}}), the fourth passes on main and
guards the success path of the fix. The camel-azure-storage-blob unit tests
pass with the fix (68 run, 0 failures).
*How it was found*: deterministic simulation testing with a fake Azure storage
world (the consumer, its scheduled polls, its on completion and Camel's routing
real; the sync {{BlobServiceClient}} / {{BlobContainerClient}} / {{BlobClient}}
faked from the Blob service REST reference, with asynchronous copies whose
final status the seed picks). The seed picked {{moveAfterRead}} or
{{deleteAfterRead}}, {{maxResultsPerPage}}, the poll delay, the {{timeout}}, an
asynchronous route step, fault rates, and how often a copy stays pending and
ends as failed, aborted or with a failed status read. The invariant "nothing
lost: every blob is still in the source container or was moved with its
content" failed in 99 of 1000 seeds on main, all with {{moveAfterRead=true}}
and a copy that did not succeed (first failing seed 11: a pending copy ended as
failed after about 4 s and the source blob was deleted); with the fix, 0 of the
same 1000 seeds fail. The world's self-check: with CAMEL-25551 reverted, 733 of
the 1000 seeds fail its invariant "a blob is not routed again while its
exchange is in flight".
*Affected*: main, camel-4.23.x and camel-4.22.x (the same
{{beginCopy(...).waitForCompletion(timeout)}} followed by the delete; read, not
run). camel-4.18.x uses {{copyFromUrl}} and is not affected.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
