[
https://issues.apache.org/jira/browse/CAMEL-25785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen updated CAMEL-25785:
--------------------------------
Fix Version/s: 4.24.0
> camel-aws2-s3 - streaming upload with timestampGroupingEnabled replaces the
> object of a time window when the window is uploaded again
> (batchMessageNumber, batchSize, streamingUploadTimeout), so the earlier
> messages of the window are lost
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25785
> URL: https://issues.apache.org/jira/browse/CAMEL-25785
> Project: Camel
> Issue Type: Bug
> Components: camel-aws2
> Reporter: shashank
> Assignee: shashank
> Priority: Major
> Fix For: 4.24.0
>
>
> {{AWS2S3StreamUploadProducer.processWithTimestampGrouping}} (main
> 7aae27315946, lines 336-450) names the object of a time window with
> {{generateTimestampBasedFileName(fileName, extension, timestampWindow)}},
> which depends only on the window:
> {{grouped-messages_20240101_0800_0800-0805.txt}}. The upload of a window is
> completed, and the window removed from {{timestampBasedUploads}}, when it
> reaches {{batchMessageNumber}} messages (default 10) or {{batchSize}} bytes
> (lines 418-436), and by the {{streamingUploadTimeout}} task (lines 133-170).
> The next message of the same window then starts a new multipart upload with
> the *same key*, and when that upload completes S3 replaces the first object
> with it:
> {code:java}
> if (!timestampBasedUploads.containsKey(timestampWindow)) {
> UploadState newState = new UploadState();
> ...
> newState.dynamicKeyName = generateTimestampBasedFileName(fileName,
> extension, timestampWindow);
> {code}
> So every message of the earlier uploads of the window is lost (unless the
> bucket is versioned), and every exchange completed normally with the ETag of
> an object that is replaced later. The example of the component documentation
> (aws2-s3-streaming.adoc, "Timestamp Grouping": a timer every 10 seconds,
> {{timestampWindowSizeMillis=300000}}, default {{batchMessageNumber}}) sends
> 30 messages per window: the window is uploaded after the 10th, 20th and 30th
> message, and the bucket keeps only the last 10. The documentation says the
> feature writes the messages of a window "to the same S3 file"; it says
> nothing of objects being replaced. The progressive, random and timestamp
> naming strategies of the non-grouped mode do give each upload its own key.
> *Fix*: the producer counts the completed uploads of each window (under its
> lock; an access-ordered map bounded to the last 10000 windows), and a later
> upload of a window gets that count after the time range:
> {{grouped-messages_20240101_0800_0800-0805-1.txt}}, {{-2}}, ... The first
> upload of a window keeps its name. Documentation of the naming in
> aws2-s3-streaming.adoc, and a note in the 4.24 upgrade guide (the key of the
> second and later objects of a window is new).
> *Tests*: new {{AWS2S3StreamUploadTimestampWindowOverwriteTest}} (mocked
> {{S3Client}} that keeps completed objects by key, like a bucket).
> {{everyMessageOfAWindowIsStoredWhenTheWindowIsUploadedMoreThanOnce}} (the
> documentation example, 30 messages in one window) fails on main: {{each
> upload of the window must be its own object, but the bucket has
> [grouped-messages_20240101_0800_0800-0805.txt] ==> expected: <3> but was:
> <1>}}.
> {{everyMessageOfAWindowIsStoredWhenTheTimeoutUploadsTheWindowBeforeItEnds}}
> (the timeout task runs in the middle of a window) fails on main: {{expected:
> <2> but was: <1>}}. {{theFirstUploadOfAWindowKeepsItsName}} passes on main
> and with the fix. {{AWS2S3StreamUploadProducerTimestampTest}} is adapted to
> the new signature of the name method and checks the suffix. camel-aws2-s3
> unit tests: 107 run, 0 failures.
> *How it was found*: variant analysis of CAMEL-25781 (pattern: a batch window
> completed by a size or count limit or a timer while the window is still open;
> what happens to the next item of that window), then the JUnit test.
> *Affected*: main, camel-4.23.x, camel-4.22.x and camel-4.18.x (the same
> naming in {{processWithTimestampGrouping}}; read, not run). Not run against
> S3.
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
