shashank created CAMEL-25770:
--------------------------------
Summary: camel-azure-storage-blob, camel-azure-storage-datalake,
camel-google-storage, camel-minio - one listed object that cannot be got
(access denied, archived) fails every poll and the consumer never consumes the
bucket again
Key: CAMEL-25770
URL: https://issues.apache.org/jira/browse/CAMEL-25770
Project: Camel
Issue Type: Bug
Components: camel-azure, camel-google-storage, camel-minio
Reporter: shashank
Assignee: shashank
The same defect as CAMEL-25558 (camel-aws2-s3) and CAMEL-25760
(camel-alibaba-oss, camel-huaweicloud-obs) in four more object-store consumers.
Each lists the whole bucket / container / file system at every poll, so one
listed item whose get fails every time fails every poll and the consumer is
stuck for good. The fix in each module mirrors CAMEL-25558: skip and report the
item, fail the poll only when none of the listed items could be got. One pull
request per module.
h3. camel-azure-storage-blob
{{BlobConsumer.poll}} (main e13303ef1ef7) lists the whole container at every
poll ({{createBatchExchangesFromContainer}}, lines 124-145; {{listBlobs}}
streams every page) and gets every listed blob in {{addExchange}} (lines
148-168; the get is {{BlobOperations.getBlob}}, or the download to
{{fileDir}}). A blob that no longer exists (404) is skipped since CAMEL-25551;
any other failure of the get is rethrown, and only a 404 of the container is
turned into an empty poll (line 93):
{code:java}
try {
for (BlobItem blobItem : blobs) {
addExchange(blobItem.getName(), blobContainerClient, exchanges, true);
}
} catch (Exception e) {
// none of the blobs of this poll is processed
releaseNotProcessed(exchanges);
throw e;
}
{code}
As the next poll lists the whole container again, one blob whose get fails
every time stops the consumer from consuming any blob, for good. The most
common case is a blob in the archive tier: a lifecycle management rule moves
old blobs to the archive tier, and reading an archived blob fails with 409
BlobArchived until it is rehydrated. Others: access denied for that blob (an
ABAC condition on blob index tags), a blob encrypted with a customer-provided
key. The consumer keeps running and logs one WARN per poll.
camel-aws2-s3 had the same behaviour and CAMEL-25558 changed it: a listed
object whose get fails is skipped for that poll and reported to the exception
handler with its key, the other listed objects are consumed, and the poll fails
only when none of the listed objects could be got. CAMEL-25760 does the same
for camel-alibaba-oss and camel-huaweicloud-obs.
*Fix*: the same in {{BlobConsumer.createBatchExchangesFromContainer}}: a failed
get of one listed blob is collected, the blob is skipped and reported to the
consumer's exception handler ("Error getting blob <name> from container
<container>. The blob is skipped for this poll."), and the other blobs are
consumed; when none of the listed blobs could be got, the poll fails with the
first failure as before. A failure with {{blobName}} set, and a failure of the
listing, are not changed. Upgrade note in the 4.24 guide.
*Tests*: new {{BlobConsumerBlobThatCannotBeGotTest}} (2 tests, the mocked
{{BlobServiceClient}} of {{BlobConsumerInProgressTest}}; the get of an archived
blob throws a {{BlobStorageException}} with status 409).
{{blobThatCannotBeGotDoesNotBlockTheOtherBlobs}} (a.txt, b.txt archived, c.txt)
fails on main: {{mock://result Received message count. Expected: <2> but was:
<0>}}. {{pollFailsWhenNoListedBlobCanBeGot}} passes on main and with the fix;
it guards the case the fix must not swallow. camel-azure-storage-blob unit
tests: 66 run, 0 failures.
*How it was found*: variant analysis of CAMEL-25558 (the fix of camel-aws2-s3)
across the object-store consumers, rule {{listing-poll-rethrows-one-item}} (a
Semgrep rule: a {{ScheduledBatchPollingConsumer}} that gets every listed item
inside one try block and rethrows the first failure). The JUnit tests above are
the evidence; the 409 BlobArchived answer is from the Azure Storage
documentation and was not run against Azure.
*Affected*: main and camel-4.23.x (the same rethrow); camel-4.22.x and
camel-4.18.x have no per-blob catch at all, so there a blob deleted after the
listing fails the poll too (read, not run).
h3. camel-azure-storage-datalake
{{DataLakeConsumer.poll}} (main e13303ef1ef7, lines 54-79) lists the whole file
system at every poll ({{createBatchExchangesFromPath}}, lines 82-101) and gets
every listed file with {{createExchangeFromFile}} (lines 103-122; the get is
{{DataLakeFileOperations.getFile}}, or the download to {{fileDir}}). Nothing
catches the failure of one file:
{code:java}
for (PathItem pathItem : items) {
if (!pathItem.isDirectory()) {
exchanges.add(createExchangeFromFile(pathItem.getName(),
dataLakeFileSystemClient));
}
}
{code}
A {{DataLakeStorageException}} with status 404 (a file deleted after the
listing) ends the whole poll with no exchange (lines 71-77; the exchanges
already created for the files listed before it are dropped, not routed), and
any other failure fails the poll. As the next poll lists the file system again,
one file whose get fails every time stops the consumer from consuming any file,
for good. In Data Lake Storage Gen2 every file has its own POSIX ACL, so a file
the consumer's identity may list but not read (403
AuthorizationPermissionMismatch) is a normal state, as is a file encrypted with
a customer-provided key. The consumer keeps running and logs one WARN per poll.
camel-aws2-s3 had the same behaviour and CAMEL-25558 changed it: a listed
object whose get fails is skipped for that poll and reported to the exception
handler with its key, the other listed objects are consumed, and the poll fails
only when none of the listed objects could be got. CAMEL-25760 does the same
for camel-alibaba-oss and camel-huaweicloud-obs.
*Fix*: the same in {{DataLakeConsumer.createBatchExchangesFromPath}}: a failed
get of one listed file is collected, the file is skipped and reported to the
consumer's exception handler ("Error getting file <name> from file system <fs>.
The file is skipped for this poll."), and the other files are consumed; a file
deleted after the listing (404) is skipped without failing the poll; when none
of the listed files could be got, the poll fails with the first failure as
before. A failure with {{fileName}} set, and a failure of the listing, are not
changed. Upgrade note in the 4.24 guide.
*Tests*: new {{DataLakeConsumerFileThatCannotBeGotTest}} (3 tests, a mocked
{{DataLakeServiceClient}}; the read of a forbidden file throws a
{{DataLakeStorageException}} with status 403).
{{fileThatCannotBeGotDoesNotBlockTheOtherFiles}} (a.txt, b.txt forbidden,
c.txt) fails on main: {{mock://result Received message count 0, expected at
least 2}}. {{fileDeletedAfterTheListingIsSkipped}} (a.txt, b.txt answering 404,
c.txt; nothing is reported) fails on main the same way.
{{pollFailsWhenNoListedFileCanBeGot}} passes on main and with the fix; it
guards the case the fix must not swallow. camel-azure-storage-datalake unit
tests: 19 run, 0 failures.
*How it was found*: variant analysis of CAMEL-25558 (the fix of camel-aws2-s3)
across the object-store consumers, rule {{listing-poll-uncaught-item-get}} (a
Semgrep rule: a {{ScheduledBatchPollingConsumer}} that creates the exchange of
every listed item, which downloads it, with no per-item catch). The JUnit tests
above are the evidence; the 403 answer is from the Azure Storage documentation
and was not run against Azure.
*Affected*: main, camel-4.23.x, camel-4.22.x and camel-4.18.x (the same loop;
read, not run).
h3. camel-google-storage
{{GoogleCloudStorageConsumer.poll}} (main e13303ef1ef7, lines 122-151) lists
the whole bucket at every poll ({{page.iterateAll()}}) and downloads every
listed object in {{createExchanges}} / {{addExchange}} (lines 160-209; the
download is {{blob.downloadTo}} in {{populateExchange}}, lines 392 and 401). An
object that no longer exists (404) is skipped since CAMEL-25519, and an object
whose name is rejected as a local path is skipped; any other failure of the
download is rethrown and fails the poll:
{code:java}
} catch (Exception e) {
LOG.warn("Error getting object due: {}", e.getMessage(), e);
// none of the objects of this poll is processed
releaseInProgress(answer);
throw e;
}
{code}
As the next poll lists the whole bucket again, one object whose download fails
every time (access denied for that object, an object encrypted with a
customer-supplied key (CSEK), an object whose Cloud KMS key version was
disabled) stops the consumer from consuming any object of the bucket, for good.
The consumer keeps running and logs a WARN with a stack trace per poll, until
someone removes the object.
camel-aws2-s3 had the same behaviour and CAMEL-25558 changed it: a listed
object whose get fails is skipped for that poll and reported to the exception
handler with its key, the other listed objects are consumed, and the poll fails
only when none of the listed objects could be got. CAMEL-25760 does the same
for camel-alibaba-oss and camel-huaweicloud-obs.
*Fix*: the same in {{GoogleCloudStorageConsumer.createExchanges}}: a failed
download of one listed object is collected, the object is skipped and reported
to the consumer's exception handler ("Error getting object <name> from bucket
<bucket>. The object is skipped for this poll."), and the other objects are
consumed; when none of the listed objects could be got, the poll fails with the
first failure as before. A failure of the listing, and a failure with
{{objectName}} set, still fail the poll. Upgrade note in the 4.24 guide.
*Tests*: new {{ConsumerObjectThatCannotBeGotTest}} (2 tests, with the module's
in-process {{LocalStorageHelper}} storage, which can now refuse the reads of an
object with 403). {{objectThatCannotBeGotDoesNotBlockTheOtherObjects}} (a.txt,
b.txt refused, c.txt) fails on main: {{mock://result Received message count.
Expected: <2> but was: <0>}}. {{pollFailsWhenNoListedObjectCanBeGot}} passes on
main and with the fix; it guards the case the fix must not swallow.
camel-google-storage unit tests: 63 run, 0 failures.
*How it was found*: variant analysis of CAMEL-25558 (the fix of camel-aws2-s3)
across the object-store consumers, rule {{listing-poll-rethrows-one-item}} (a
Semgrep rule: a {{ScheduledBatchPollingConsumer}} that gets every listed item
inside one try block and rethrows the first failure). The JUnit tests above are
the evidence; the status codes of Cloud Storage were not run against the
service.
*Affected*: main, camel-4.23.x, camel-4.22.x and camel-4.18.x (the same rethrow
in {{createExchanges}}; read, not run).
h3. camel-minio
{{MinioConsumer}} (main e13303ef1ef7) stats every listed object when it creates
the exchanges ({{createExchanges}} / {{addExchange}}, lines 201-254, the stat
is {{MinioEndpoint.getObjectStat}}), and gets the body of each object when it
routes the batch ({{processBatch}}, lines 289-331). An object that no longer
exists is skipped since CAMEL-25512. Any other failure is rethrown:
{code:java}
} catch (Exception e) {
...
LOG.warn("Error getting MinioObject due: {}", e.getMessage());
// this and the remaining exchanges of the batch are not processed
releaseInProgress(srcObjectName, exchanges);
throw e;
}
{code}
So one object whose stat or get fails every time (access denied for that object
by a bucket or IAM policy, an object encrypted with a customer key (SSE-C)
other than the one of the endpoint) fails every poll: a failed stat drops every
exchange of the poll, and a failed get stops the batch, so the objects listed
after it are not consumed. The next poll lists the same objects again (with the
default {{deleteAfterRead=true}} the objects before it are gone, so the bad
object comes first and nothing after it is ever consumed). The consumer keeps
running and logs a WARN at every poll; it lasts until someone removes the
object.
camel-aws2-s3 had the same behaviour and CAMEL-25558 changed it: a listed
object whose get fails is skipped for that poll and reported to the exception
handler with its key, the other listed objects are consumed, and the poll fails
only when none of the listed objects could be got. CAMEL-25760 does the same
for camel-alibaba-oss and camel-huaweicloud-obs.
*Fix*: the same in {{MinioConsumer}}, for both places: a failed stat (in
{{createExchanges}}) or get (in {{processBatch}}) of one listed object is
collected, the object is skipped (released from the in-progress set) and
reported to the consumer's exception handler ("Error getting object <name> from
bucket <bucket>. The object is skipped for this poll."), and the other objects
are consumed; when none of the objects could be got, the poll fails with the
first failure as before. A failure of the listing itself, and a failure with
{{objectName}} set, still fail the poll. Upgrade note in the 4.24 guide.
*Tests*: new {{MinioConsumerObjectThatCannotBeGotTest}} (3 tests, the real
Minio client against the in-process fake S3 server of the module's tests, which
can now refuse the stats and gets of an object with 403 AccessDenied).
{{objectWhoseBodyCannotBeGotDoesNotBlockTheOtherObjects}} (a.txt, b.txt whose
get is refused, c.txt) fails on main: {{mock://result Received message count.
Expected: <2> but was: <1>}} (c.txt is never consumed).
{{objectThatCannotBeStatedDoesNotBlockTheOtherObjects}} fails on main:
{{Expected: <2> but was: <0>}}. {{pollFailsWhenNoListedObjectCanBeGot}} passes
on main and with the fix; it guards the case the fix must not swallow.
camel-minio unit tests: 10 run, 0 failures.
*How it was found*: variant analysis of CAMEL-25558 (the fix of camel-aws2-s3)
across the object-store consumers, rule {{listing-poll-rethrows-one-item}} (a
Semgrep rule: a {{ScheduledBatchPollingConsumer}} that gets every listed item
inside one try block and rethrows the first failure). The JUnit tests above are
the evidence; the S3 status codes are those of the S3 API and were not run
against a MinIO server.
*Affected*: main, camel-4.23.x, camel-4.22.x and camel-4.18.x (the same rethrow
in {{createExchanges}} and {{processBatch}}; on 4.22.x and 4.18.x, which do not
have CAMEL-25512, an object deleted after the listing fails the poll too; read,
not run). The component is deprecated since 4.23 (CAMEL-24750) but still
shipped; CAMEL-25512 was fixed in it after the deprecation.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
