[
https://issues.apache.org/jira/browse/CAMEL-25760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen updated CAMEL-25760:
--------------------------------
Fix Version/s: 4.24.0
> camel-alibaba-oss, camel-huaweicloud-obs - one listed object that cannot be
> got (access denied, archived) fails every poll, so the consumer stops
> consuming the bucket (OSS: never again; OBS: the objects listed with it)
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25760
> URL: https://issues.apache.org/jira/browse/CAMEL-25760
> Project: Camel
> Issue Type: Bug
> Reporter: shashank
> Assignee: shashank
> Priority: Major
> Fix For: 4.24.0
>
>
> The same defect in two object-store consumers (as CAMEL-25551 covered several
> modules in one issue); one fix per module, two pull requests.
> h3. camel-alibaba-oss
> {{OSSConsumer.poll}} (main e13303ef1ef7, lines 72-107) lists one page of the
> bucket ({{maxKeys}} = {{maxMessagesPerPoll}}, 10 by default) and gets every
> listed object in {{createExchanges}} / {{addExchange}} (lines 152-203). An
> object that no longer exists (404) is skipped since CAMEL-25551; any other
> failure of the get is rethrown, the exchanges already created are released,
> and the poll fails. The continuation token is only moved on after the batch
> was processed (lines 101-105), so the next poll lists the same page again:
> {code:java}
> Queue<Exchange> exchanges = createExchanges(bucketName, listing.contents());
> int processed = processBatch(CastUtils.cast(exchanges));
> if (processed >= 0) {
> continuationToken = nextToken;
> }
> {code}
> So one object whose get fails every time (access denied for that object, an
> object of the Archive or Cold Archive storage class that is not restored, a
> KMS key that was disabled) fails every poll: none of the other objects of its
> page is consumed, and the consumer never reaches the following pages. If the
> bucket fits in one page, the consumer consumes nothing at all. Nothing but a
> WARN per poll ("Failed polling endpoint") says why, and it lasts until
> someone removes or restores the object.
> camel-aws2-s3 had the same behaviour and CAMEL-25558 changed it: a listed
> object whose get fails is skipped for that poll and reported to the exception
> handler with its key, the other listed objects are consumed, and the poll
> fails only when none of the listed objects could be got (expired credentials,
> no permission to get objects); the marker of the listing is moved on before
> the objects are got ({{AWS2S3Consumer}} lines 173-201, 340-356).
> camel-huaweicloud-obs moves its marker on before the gets, but also fails the
> poll for one object.
> *Fix*: the same in {{OSSConsumer}}: a failed get of one listed object is
> collected, the object is skipped and reported to the consumer's exception
> handler ("Error getting object <key> from bucket <bucket>. The object is
> skipped for this poll."), and the other objects are consumed; when none of
> the listed objects could be got, the poll fails with the first failure as
> before. The continuation token is set from the listing before the objects are
> got, so a page whose objects cannot be got does not hold the consumer back
> (its objects are listed again when the listing starts over). An object
> deleted after the listing is still skipped silently. Upgrade note in the 4.24
> guide.
> *Tests*: new {{OSSConsumerObjectThatCannotBeGotTest}} (3 tests; a fake bucket
> that lists keys in order, {{maxKeys}} at a time with a continuation token,
> and answers the get of a forbidden object with 403).
> {{objectThatCannotBeGotDoesNotBlockTheOtherObjects}} (a.txt, b.txt forbidden,
> c.txt) and {{pageOfObjectsThatCannotBeGotDoesNotBlockTheNextPage}}
> ({{maxMessagesPerPoll=1}}, a.txt forbidden) fail on main: no object is
> received ({{mock://result Received message count. Expected: <2> but was:
> <0>}}). {{pollFailsWhenNoListedObjectCanBeGot}} passes on main and with the
> fix; it guards the case the fix must not swallow (no object can be got: the
> poll fails). camel-alibaba-oss unit tests: 22 run, 0 failures.
> *How it was found*: differential reading of sibling consumers of object
> stores against the same listing / get / commit model: camel-aws2-s3 (after
> CAMEL-25558), camel-huaweicloud-obs and camel-alibaba-oss. camel-aws2-s3
> skips the object; camel-huaweicloud-obs fails the poll but moves its marker
> on, so it stops for good only when the bucket fits in one page (see below);
> camel-alibaba-oss fails the poll and keeps its listing position, so it stops
> for good whatever the size of the bucket. The JUnit tests above are the
> evidence; the OSS status codes for archived and forbidden objects are from
> the OSS documentation and were not run against OSS.
> Not changed here: camel-minio (a failed get in
> {{MinioConsumer.processBatch}}, main lines 319-331, stops the rest of the
> batch and fails the poll) and camel-google-storage
> ({{GoogleCloudStorageConsumer.createExchanges}} / {{addExchange}}, lines
> 166-208, fail the poll for one object) have the same kind of defect; they are
> left as follow-ups.
> *Affected*: main, camel-4.23.x (the same lines; read, not run). The component
> is new in 4.23.
> h3. camel-huaweicloud-obs
> {{OBSConsumer.poll}} (main e13303ef1ef7, lines 139-166) lists the bucket
> ({{maxKeys}} = {{maxMessagesPerPoll}}, 10 by default) and gets every listed
> object in {{createExchanges}} (lines 218-245). An object that no longer
> exists (404) is skipped since CAMEL-25551; any other failure of the get is
> rethrown, the exchanges already created are released, and the poll fails:
> {code:java}
> } catch (ObsException e) {
> if (e.getResponseCode() == 404) { ... continue; }
> throw e;
> }
> ...
> } catch (RuntimeException e) {
> // none of the objects of this poll is processed
> releaseNotProcessed(answer);
> throw e;
> }
> {code}
> So one object whose get fails every time (access denied for that object, an
> object of the Archive or Deep Archive storage class that is not restored, an
> SSE-KMS key that was disabled) fails every poll that lists it, and the other
> objects of that listing are not consumed either. The marker of a truncated
> listing is moved on before the gets, but a listing that is not truncated sets
> the marker back to {{null}}, so:
> * a bucket (or prefix) whose objects fit in one listing (10 objects with the
> default {{maxMessagesPerPoll}}) is never consumed again: every poll lists the
> same objects and fails on the same one;
> * in a larger bucket the objects that are listed together with it are skipped
> at every pass over the bucket.
> Nothing but a WARN per poll ("Failed polling endpoint") says why, and it
> lasts until someone removes or restores the object.
> camel-aws2-s3 had the same behaviour and CAMEL-25558 changed it: a listed
> object whose get fails is skipped for that poll and reported to the exception
> handler with its key, the other listed objects are consumed, and the poll
> fails only when none of the listed objects could be got ({{AWS2S3Consumer}}
> lines 340-356).
> *Fix*: the same in {{OBSConsumer.createExchanges}}: a failed get of one
> listed object is collected, the object is skipped and reported to the
> consumer's exception handler ("Error getting object <key> from bucket
> <bucket>. The object is skipped for this poll."), and the other objects are
> consumed; when none of the listed objects could be got, the poll fails with
> the first failure as before. An object deleted after the listing is still
> skipped silently; with the {{fileName}} option a failed get still fails the
> poll. Upgrade note in the 4.24 guide.
> *Tests*: new {{OBSConsumerObjectThatCannotBeGotTest}} (3 tests; a fake bucket
> that lists keys in order, {{maxKeys}} at a time with the last key as next
> marker, and answers the get of a forbidden object with 403).
> {{objectThatCannotBeGotDoesNotBlockTheOtherObjects}} (a.txt, b.txt forbidden,
> c.txt, one listing) and
> {{objectThatCannotBeGotDoesNotBlockTheObjectsOfItsPage}}
> ({{maxMessagesPerPoll=2}}, a.txt forbidden, listed with b.txt) fail on main.
> {{pollFailsWhenNoListedObjectCanBeGot}} passes on main and with the fix; it
> guards the case the fix must not swallow (no object can be got: the poll
> fails). camel-huaweicloud-obs unit tests: 47 run, 0 failures, 18 skipped
> (functional tests that need OBS credentials).
> *How it was found*: differential reading of sibling consumers of object
> stores against the same listing / get / commit model: camel-aws2-s3 (after
> CAMEL-25558), camel-huaweicloud-obs and camel-alibaba-oss. camel-aws2-s3
> skips the object; camel-huaweicloud-obs and camel-alibaba-oss fail the poll.
> The JUnit tests above are the evidence; the OBS status codes for archived and
> forbidden objects are from the OBS documentation and were not run against OBS.
> Not changed here: camel-minio (a failed get in
> {{MinioConsumer.processBatch}}, main lines 319-331, stops the rest of the
> batch and fails the poll) and camel-google-storage
> ({{GoogleCloudStorageConsumer.createExchanges}} / {{addExchange}}, lines
> 166-208, fail the poll for one object) have the same kind of defect; they are
> left as follow-ups.
> *Affected*: main and camel-4.23.x (the same lines); camel-4.22.x,
> camel-4.18.x and camel-4.14.x do not catch the get at all, so there every
> failed get, also of an object deleted after the listing, fails the poll
> (read, not run).
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
