[
https://issues.apache.org/jira/browse/CAMEL-25770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen resolved CAMEL-25770.
---------------------------------
Resolution: Fixed
> camel-azure-storage-blob, camel-azure-storage-datalake, camel-google-storage,
> camel-minio - one listed object that cannot be got (access denied, archived)
> fails every poll and the consumer never consumes the bucket again
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25770
> URL: https://issues.apache.org/jira/browse/CAMEL-25770
> Project: Camel
> Issue Type: Bug
> Components: camel-azure, camel-google-storage, camel-minio
> Reporter: shashank
> Assignee: shashank
> Priority: Major
> Fix For: 4.24.0
>
>
> The same defect as CAMEL-25558 (camel-aws2-s3) and CAMEL-25760
> (camel-alibaba-oss, camel-huaweicloud-obs) in four more object-store
> consumers. Each lists the whole bucket / container / file system at every
> poll, so one listed item whose get fails every time fails every poll and the
> consumer is stuck for good. The fix in each module mirrors CAMEL-25558: skip
> and report the item, fail the poll only when none of the listed items could
> be got. One pull request per module.
> h3. camel-azure-storage-blob
> {{BlobConsumer.poll}} (main e13303ef1ef7) lists the whole container at every
> poll ({{createBatchExchangesFromContainer}}, lines 124-145; {{listBlobs}}
> streams every page) and gets every listed blob in {{addExchange}} (lines
> 148-168; the get is {{BlobOperations.getBlob}}, or the download to
> {{fileDir}}). A blob that no longer exists (404) is skipped since
> CAMEL-25551; any other failure of the get is rethrown, and only a 404 of the
> container is turned into an empty poll (line 93):
> {code:java}
> try {
> for (BlobItem blobItem : blobs) {
> addExchange(blobItem.getName(), blobContainerClient, exchanges, true);
> }
> } catch (Exception e) {
> // none of the blobs of this poll is processed
> releaseNotProcessed(exchanges);
> throw e;
> }
> {code}
> As the next poll lists the whole container again, one blob whose get fails
> every time stops the consumer from consuming any blob, for good. The most
> common case is a blob in the archive tier: a lifecycle management rule moves
> old blobs to the archive tier, and reading an archived blob fails with 409
> BlobArchived until it is rehydrated. Others: access denied for that blob (an
> ABAC condition on blob index tags), a blob encrypted with a customer-provided
> key. The consumer keeps running and logs one WARN per poll.
> camel-aws2-s3 had the same behaviour and CAMEL-25558 changed it: a listed
> object whose get fails is skipped for that poll and reported to the exception
> handler with its key, the other listed objects are consumed, and the poll
> fails only when none of the listed objects could be got. CAMEL-25760 does the
> same for camel-alibaba-oss and camel-huaweicloud-obs.
> *Fix*: the same in {{BlobConsumer.createBatchExchangesFromContainer}}: a
> failed get of one listed blob is collected, the blob is skipped and reported
> to the consumer's exception handler ("Error getting blob <name> from
> container <container>. The blob is skipped for this poll."), and the other
> blobs are consumed; when none of the listed blobs could be got, the poll
> fails with the first failure as before. A failure with {{blobName}} set, and
> a failure of the listing, are not changed. Upgrade note in the 4.24 guide.
> *Tests*: new {{BlobConsumerBlobThatCannotBeGotTest}} (2 tests, the mocked
> {{BlobServiceClient}} of {{BlobConsumerInProgressTest}}; the get of an
> archived blob throws a {{BlobStorageException}} with status 409).
> {{blobThatCannotBeGotDoesNotBlockTheOtherBlobs}} (a.txt, b.txt archived,
> c.txt) fails on main: {{mock://result Received message count. Expected: <2>
> but was: <0>}}. {{pollFailsWhenNoListedBlobCanBeGot}} passes on main and with
> the fix; it guards the case the fix must not swallow.
> camel-azure-storage-blob unit tests: 66 run, 0 failures.
> *How it was found*: variant analysis of CAMEL-25558 (the fix of
> camel-aws2-s3) across the object-store consumers, rule
> {{listing-poll-rethrows-one-item}} (a Semgrep rule: a
> {{ScheduledBatchPollingConsumer}} that gets every listed item inside one try
> block and rethrows the first failure). The JUnit tests above are the
> evidence; the 409 BlobArchived answer is from the Azure Storage documentation
> and was not run against Azure.
> *Affected*: main and camel-4.23.x (the same rethrow); camel-4.22.x and
> camel-4.18.x have no per-blob catch at all, so there a blob deleted after the
> listing fails the poll too (read, not run).
> h3. camel-azure-storage-datalake
> {{DataLakeConsumer.poll}} (main e13303ef1ef7, lines 54-79) lists the whole
> file system at every poll ({{createBatchExchangesFromPath}}, lines 82-101)
> and gets every listed file with {{createExchangeFromFile}} (lines 103-122;
> the get is {{DataLakeFileOperations.getFile}}, or the download to
> {{fileDir}}). Nothing catches the failure of one file:
> {code:java}
> for (PathItem pathItem : items) {
> if (!pathItem.isDirectory()) {
> exchanges.add(createExchangeFromFile(pathItem.getName(),
> dataLakeFileSystemClient));
> }
> }
> {code}
> A {{DataLakeStorageException}} with status 404 (a file deleted after the
> listing) ends the whole poll with no exchange (lines 71-77; the exchanges
> already created for the files listed before it are dropped, not routed), and
> any other failure fails the poll. As the next poll lists the file system
> again, one file whose get fails every time stops the consumer from consuming
> any file, for good. In Data Lake Storage Gen2 every file has its own POSIX
> ACL, so a file the consumer's identity may list but not read (403
> AuthorizationPermissionMismatch) is a normal state, as is a file encrypted
> with a customer-provided key. The consumer keeps running and logs one WARN
> per poll.
> camel-aws2-s3 had the same behaviour and CAMEL-25558 changed it: a listed
> object whose get fails is skipped for that poll and reported to the exception
> handler with its key, the other listed objects are consumed, and the poll
> fails only when none of the listed objects could be got. CAMEL-25760 does the
> same for camel-alibaba-oss and camel-huaweicloud-obs.
> *Fix*: the same in {{DataLakeConsumer.createBatchExchangesFromPath}}: a
> failed get of one listed file is collected, the file is skipped and reported
> to the consumer's exception handler ("Error getting file <name> from file
> system <fs>. The file is skipped for this poll."), and the other files are
> consumed; a file deleted after the listing (404) is skipped without failing
> the poll; when none of the listed files could be got, the poll fails with the
> first failure as before. A failure with {{fileName}} set, and a failure of
> the listing, are not changed. Upgrade note in the 4.24 guide.
> *Tests*: new {{DataLakeConsumerFileThatCannotBeGotTest}} (3 tests, a mocked
> {{DataLakeServiceClient}}; the read of a forbidden file throws a
> {{DataLakeStorageException}} with status 403).
> {{fileThatCannotBeGotDoesNotBlockTheOtherFiles}} (a.txt, b.txt forbidden,
> c.txt) fails on main: {{mock://result Received message count 0, expected at
> least 2}}. {{fileDeletedAfterTheListingIsSkipped}} (a.txt, b.txt answering
> 404, c.txt; nothing is reported) fails on main the same way.
> {{pollFailsWhenNoListedFileCanBeGot}} passes on main and with the fix; it
> guards the case the fix must not swallow. camel-azure-storage-datalake unit
> tests: 19 run, 0 failures.
> *How it was found*: variant analysis of CAMEL-25558 (the fix of
> camel-aws2-s3) across the object-store consumers, rule
> {{listing-poll-uncaught-item-get}} (a Semgrep rule: a
> {{ScheduledBatchPollingConsumer}} that creates the exchange of every listed
> item, which downloads it, with no per-item catch). The JUnit tests above are
> the evidence; the 403 answer is from the Azure Storage documentation and was
> not run against Azure.
> *Affected*: main, camel-4.23.x, camel-4.22.x and camel-4.18.x (the same loop;
> read, not run).
> h3. camel-google-storage
> {{GoogleCloudStorageConsumer.poll}} (main e13303ef1ef7, lines 122-151) lists
> the whole bucket at every poll ({{page.iterateAll()}}) and downloads every
> listed object in {{createExchanges}} / {{addExchange}} (lines 160-209; the
> download is {{blob.downloadTo}} in {{populateExchange}}, lines 392 and 401).
> An object that no longer exists (404) is skipped since CAMEL-25519, and an
> object whose name is rejected as a local path is skipped; any other failure
> of the download is rethrown and fails the poll:
> {code:java}
> } catch (Exception e) {
> LOG.warn("Error getting object due: {}", e.getMessage(), e);
> // none of the objects of this poll is processed
> releaseInProgress(answer);
> throw e;
> }
> {code}
> As the next poll lists the whole bucket again, one object whose download
> fails every time (access denied for that object, an object encrypted with a
> customer-supplied key (CSEK), an object whose Cloud KMS key version was
> disabled) stops the consumer from consuming any object of the bucket, for
> good. The consumer keeps running and logs a WARN with a stack trace per poll,
> until someone removes the object.
> camel-aws2-s3 had the same behaviour and CAMEL-25558 changed it: a listed
> object whose get fails is skipped for that poll and reported to the exception
> handler with its key, the other listed objects are consumed, and the poll
> fails only when none of the listed objects could be got. CAMEL-25760 does the
> same for camel-alibaba-oss and camel-huaweicloud-obs.
> *Fix*: the same in {{GoogleCloudStorageConsumer.createExchanges}}: a failed
> download of one listed object is collected, the object is skipped and
> reported to the consumer's exception handler ("Error getting object <name>
> from bucket <bucket>. The object is skipped for this poll."), and the other
> objects are consumed; when none of the listed objects could be got, the poll
> fails with the first failure as before. A failure of the listing, and a
> failure with {{objectName}} set, still fail the poll. Upgrade note in the
> 4.24 guide.
> *Tests*: new {{ConsumerObjectThatCannotBeGotTest}} (2 tests, with the
> module's in-process {{LocalStorageHelper}} storage, which can now refuse the
> reads of an object with 403).
> {{objectThatCannotBeGotDoesNotBlockTheOtherObjects}} (a.txt, b.txt refused,
> c.txt) fails on main: {{mock://result Received message count. Expected: <2>
> but was: <0>}}. {{pollFailsWhenNoListedObjectCanBeGot}} passes on main and
> with the fix; it guards the case the fix must not swallow.
> camel-google-storage unit tests: 63 run, 0 failures.
> *How it was found*: variant analysis of CAMEL-25558 (the fix of
> camel-aws2-s3) across the object-store consumers, rule
> {{listing-poll-rethrows-one-item}} (a Semgrep rule: a
> {{ScheduledBatchPollingConsumer}} that gets every listed item inside one try
> block and rethrows the first failure). The JUnit tests above are the
> evidence; the status codes of Cloud Storage were not run against the service.
> *Affected*: main, camel-4.23.x, camel-4.22.x and camel-4.18.x (the same
> rethrow in {{createExchanges}}; read, not run).
> h3. camel-minio
> {{MinioConsumer}} (main e13303ef1ef7) stats every listed object when it
> creates the exchanges ({{createExchanges}} / {{addExchange}}, lines 201-254,
> the stat is {{MinioEndpoint.getObjectStat}}), and gets the body of each
> object when it routes the batch ({{processBatch}}, lines 289-331). An object
> that no longer exists is skipped since CAMEL-25512. Any other failure is
> rethrown:
> {code:java}
> } catch (Exception e) {
> ...
> LOG.warn("Error getting MinioObject due: {}", e.getMessage());
> // this and the remaining exchanges of the batch are not processed
> releaseInProgress(srcObjectName, exchanges);
> throw e;
> }
> {code}
> So one object whose stat or get fails every time (access denied for that
> object by a bucket or IAM policy, an object encrypted with a customer key
> (SSE-C) other than the one of the endpoint) fails every poll: a failed stat
> drops every exchange of the poll, and a failed get stops the batch, so the
> objects listed after it are not consumed. The next poll lists the same
> objects again (with the default {{deleteAfterRead=true}} the objects before
> it are gone, so the bad object comes first and nothing after it is ever
> consumed). The consumer keeps running and logs a WARN at every poll; it lasts
> until someone removes the object.
> camel-aws2-s3 had the same behaviour and CAMEL-25558 changed it: a listed
> object whose get fails is skipped for that poll and reported to the exception
> handler with its key, the other listed objects are consumed, and the poll
> fails only when none of the listed objects could be got. CAMEL-25760 does the
> same for camel-alibaba-oss and camel-huaweicloud-obs.
> *Fix*: the same in {{MinioConsumer}}, for both places: a failed stat (in
> {{createExchanges}}) or get (in {{processBatch}}) of one listed object is
> collected, the object is skipped (released from the in-progress set) and
> reported to the consumer's exception handler ("Error getting object <name>
> from bucket <bucket>. The object is skipped for this poll."), and the other
> objects are consumed; when none of the objects could be got, the poll fails
> with the first failure as before. A failure of the listing itself, and a
> failure with {{objectName}} set, still fail the poll. Upgrade note in the
> 4.24 guide.
> *Tests*: new {{MinioConsumerObjectThatCannotBeGotTest}} (3 tests, the real
> Minio client against the in-process fake S3 server of the module's tests,
> which can now refuse the stats and gets of an object with 403 AccessDenied).
> {{objectWhoseBodyCannotBeGotDoesNotBlockTheOtherObjects}} (a.txt, b.txt whose
> get is refused, c.txt) fails on main: {{mock://result Received message count.
> Expected: <2> but was: <1>}} (c.txt is never consumed).
> {{objectThatCannotBeStatedDoesNotBlockTheOtherObjects}} fails on main:
> {{Expected: <2> but was: <0>}}. {{pollFailsWhenNoListedObjectCanBeGot}}
> passes on main and with the fix; it guards the case the fix must not swallow.
> camel-minio unit tests: 10 run, 0 failures.
> *How it was found*: variant analysis of CAMEL-25558 (the fix of
> camel-aws2-s3) across the object-store consumers, rule
> {{listing-poll-rethrows-one-item}} (a Semgrep rule: a
> {{ScheduledBatchPollingConsumer}} that gets every listed item inside one try
> block and rethrows the first failure). The JUnit tests above are the
> evidence; the S3 status codes are those of the S3 API and were not run
> against a MinIO server.
> *Affected*: main, camel-4.23.x, camel-4.22.x and camel-4.18.x (the same
> rethrow in {{createExchanges}} and {{processBatch}}; on 4.22.x and 4.18.x,
> which do not have CAMEL-25512, an object deleted after the listing fails the
> poll too; read, not run). The component is deprecated since 4.23
> (CAMEL-24750) but still shipped; CAMEL-25512 was fixed in it after the
> deprecation.
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
