shashank created CAMEL-25760:
--------------------------------
Summary: camel-alibaba-oss, camel-huaweicloud-obs - one listed
object that cannot be got (access denied, archived) fails every poll, so the
consumer stops consuming the bucket (OSS: never again; OBS: the objects listed
with it)
Key: CAMEL-25760
URL: https://issues.apache.org/jira/browse/CAMEL-25760
Project: Camel
Issue Type: Bug
Reporter: shashank
Assignee: shashank
The same defect in two object-store consumers (as CAMEL-25551 covered several
modules in one issue); one fix per module, two pull requests.
h3. camel-alibaba-oss
{{OSSConsumer.poll}} (main e13303ef1ef7, lines 72-107) lists one page of the
bucket ({{maxKeys}} = {{maxMessagesPerPoll}}, 10 by default) and gets every
listed object in {{createExchanges}} / {{addExchange}} (lines 152-203). An
object that no longer exists (404) is skipped since CAMEL-25551; any other
failure of the get is rethrown, the exchanges already created are released, and
the poll fails. The continuation token is only moved on after the batch was
processed (lines 101-105), so the next poll lists the same page again:
{code:java}
Queue<Exchange> exchanges = createExchanges(bucketName, listing.contents());
int processed = processBatch(CastUtils.cast(exchanges));
if (processed >= 0) {
continuationToken = nextToken;
}
{code}
So one object whose get fails every time (access denied for that object, an
object of the Archive or Cold Archive storage class that is not restored, a KMS
key that was disabled) fails every poll: none of the other objects of its page
is consumed, and the consumer never reaches the following pages. If the bucket
fits in one page, the consumer consumes nothing at all. Nothing but a WARN per
poll ("Failed polling endpoint") says why, and it lasts until someone removes
or restores the object.
camel-aws2-s3 had the same behaviour and CAMEL-25558 changed it: a listed
object whose get fails is skipped for that poll and reported to the exception
handler with its key, the other listed objects are consumed, and the poll fails
only when none of the listed objects could be got (expired credentials, no
permission to get objects); the marker of the listing is moved on before the
objects are got ({{AWS2S3Consumer}} lines 173-201, 340-356).
camel-huaweicloud-obs moves its marker on before the gets, but also fails the
poll for one object.
*Fix*: the same in {{OSSConsumer}}: a failed get of one listed object is
collected, the object is skipped and reported to the consumer's exception
handler ("Error getting object <key> from bucket <bucket>. The object is
skipped for this poll."), and the other objects are consumed; when none of the
listed objects could be got, the poll fails with the first failure as before.
The continuation token is set from the listing before the objects are got, so a
page whose objects cannot be got does not hold the consumer back (its objects
are listed again when the listing starts over). An object deleted after the
listing is still skipped silently. Upgrade note in the 4.24 guide.
*Tests*: new {{OSSConsumerObjectThatCannotBeGotTest}} (3 tests; a fake bucket
that lists keys in order, {{maxKeys}} at a time with a continuation token, and
answers the get of a forbidden object with 403).
{{objectThatCannotBeGotDoesNotBlockTheOtherObjects}} (a.txt, b.txt forbidden,
c.txt) and {{pageOfObjectsThatCannotBeGotDoesNotBlockTheNextPage}}
({{maxMessagesPerPoll=1}}, a.txt forbidden) fail on main: no object is received
({{mock://result Received message count. Expected: <2> but was: <0>}}).
{{pollFailsWhenNoListedObjectCanBeGot}} passes on main and with the fix; it
guards the case the fix must not swallow (no object can be got: the poll
fails). camel-alibaba-oss unit tests: 22 run, 0 failures.
*How it was found*: differential reading of sibling consumers of object stores
against the same listing / get / commit model: camel-aws2-s3 (after
CAMEL-25558), camel-huaweicloud-obs and camel-alibaba-oss. camel-aws2-s3 skips
the object; camel-huaweicloud-obs fails the poll but moves its marker on, so it
stops for good only when the bucket fits in one page (see below);
camel-alibaba-oss fails the poll and keeps its listing position, so it stops
for good whatever the size of the bucket. The JUnit tests above are the
evidence; the OSS status codes for archived and forbidden objects are from the
OSS documentation and were not run against OSS.
Not changed here: camel-minio (a failed get in {{MinioConsumer.processBatch}},
main lines 319-331, stops the rest of the batch and fails the poll) and
camel-google-storage ({{GoogleCloudStorageConsumer.createExchanges}} /
{{addExchange}}, lines 166-208, fail the poll for one object) have the same
kind of defect; they are left as follow-ups.
*Affected*: main, camel-4.23.x (the same lines; read, not run). The component
is new in 4.23.
h3. camel-huaweicloud-obs
{{OBSConsumer.poll}} (main e13303ef1ef7, lines 139-166) lists the bucket
({{maxKeys}} = {{maxMessagesPerPoll}}, 10 by default) and gets every listed
object in {{createExchanges}} (lines 218-245). An object that no longer exists
(404) is skipped since CAMEL-25551; any other failure of the get is rethrown,
the exchanges already created are released, and the poll fails:
{code:java}
} catch (ObsException e) {
if (e.getResponseCode() == 404) { ... continue; }
throw e;
}
...
} catch (RuntimeException e) {
// none of the objects of this poll is processed
releaseNotProcessed(answer);
throw e;
}
{code}
So one object whose get fails every time (access denied for that object, an
object of the Archive or Deep Archive storage class that is not restored, an
SSE-KMS key that was disabled) fails every poll that lists it, and the other
objects of that listing are not consumed either. The marker of a truncated
listing is moved on before the gets, but a listing that is not truncated sets
the marker back to {{null}}, so:
* a bucket (or prefix) whose objects fit in one listing (10 objects with the
default {{maxMessagesPerPoll}}) is never consumed again: every poll lists the
same objects and fails on the same one;
* in a larger bucket the objects that are listed together with it are skipped
at every pass over the bucket.
Nothing but a WARN per poll ("Failed polling endpoint") says why, and it lasts
until someone removes or restores the object.
camel-aws2-s3 had the same behaviour and CAMEL-25558 changed it: a listed
object whose get fails is skipped for that poll and reported to the exception
handler with its key, the other listed objects are consumed, and the poll fails
only when none of the listed objects could be got ({{AWS2S3Consumer}} lines
340-356).
*Fix*: the same in {{OBSConsumer.createExchanges}}: a failed get of one listed
object is collected, the object is skipped and reported to the consumer's
exception handler ("Error getting object <key> from bucket <bucket>. The object
is skipped for this poll."), and the other objects are consumed; when none of
the listed objects could be got, the poll fails with the first failure as
before. An object deleted after the listing is still skipped silently; with the
{{fileName}} option a failed get still fails the poll. Upgrade note in the 4.24
guide.
*Tests*: new {{OBSConsumerObjectThatCannotBeGotTest}} (3 tests; a fake bucket
that lists keys in order, {{maxKeys}} at a time with the last key as next
marker, and answers the get of a forbidden object with 403).
{{objectThatCannotBeGotDoesNotBlockTheOtherObjects}} (a.txt, b.txt forbidden,
c.txt, one listing) and
{{objectThatCannotBeGotDoesNotBlockTheObjectsOfItsPage}}
({{maxMessagesPerPoll=2}}, a.txt forbidden, listed with b.txt) fail on main.
{{pollFailsWhenNoListedObjectCanBeGot}} passes on main and with the fix; it
guards the case the fix must not swallow (no object can be got: the poll
fails). camel-huaweicloud-obs unit tests: 47 run, 0 failures, 18 skipped
(functional tests that need OBS credentials).
*How it was found*: differential reading of sibling consumers of object stores
against the same listing / get / commit model: camel-aws2-s3 (after
CAMEL-25558), camel-huaweicloud-obs and camel-alibaba-oss. camel-aws2-s3 skips
the object; camel-huaweicloud-obs and camel-alibaba-oss fail the poll. The
JUnit tests above are the evidence; the OBS status codes for archived and
forbidden objects are from the OBS documentation and were not run against OBS.
Not changed here: camel-minio (a failed get in {{MinioConsumer.processBatch}},
main lines 319-331, stops the rest of the batch and fails the poll) and
camel-google-storage ({{GoogleCloudStorageConsumer.createExchanges}} /
{{addExchange}}, lines 166-208, fail the poll for one object) have the same
kind of defect; they are left as follow-ups.
*Affected*: main and camel-4.23.x (the same lines); camel-4.22.x, camel-4.18.x
and camel-4.14.x do not catch the get at all, so there every failed get, also
of an object deleted after the listing, fails the poll (read, not run).
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
