shashank created CAMEL-25785:
--------------------------------
Summary: camel-aws2-s3 - streaming upload with
timestampGroupingEnabled replaces the object of a time window when the window
is uploaded again (batchMessageNumber, batchSize, streamingUploadTimeout), so
the earlier messages of the window are lost
Key: CAMEL-25785
URL: https://issues.apache.org/jira/browse/CAMEL-25785
Project: Camel
Issue Type: Bug
Components: camel-aws2
Reporter: shashank
Assignee: shashank
{{AWS2S3StreamUploadProducer.processWithTimestampGrouping}} (main 7aae27315946,
lines 336-450) names the object of a time window with
{{generateTimestampBasedFileName(fileName, extension, timestampWindow)}}, which
depends only on the window: {{grouped-messages_20240101_0800_0800-0805.txt}}.
The upload of a window is completed, and the window removed from
{{timestampBasedUploads}}, when it reaches {{batchMessageNumber}} messages
(default 10) or {{batchSize}} bytes (lines 418-436), and by the
{{streamingUploadTimeout}} task (lines 133-170). The next message of the same
window then starts a new multipart upload with the *same key*, and when that
upload completes S3 replaces the first object with it:
{code:java}
if (!timestampBasedUploads.containsKey(timestampWindow)) {
UploadState newState = new UploadState();
...
newState.dynamicKeyName = generateTimestampBasedFileName(fileName,
extension, timestampWindow);
{code}
So every message of the earlier uploads of the window is lost (unless the
bucket is versioned), and every exchange completed normally with the ETag of an
object that is replaced later. The example of the component documentation
(aws2-s3-streaming.adoc, "Timestamp Grouping": a timer every 10 seconds,
{{timestampWindowSizeMillis=300000}}, default {{batchMessageNumber}}) sends 30
messages per window: the window is uploaded after the 10th, 20th and 30th
message, and the bucket keeps only the last 10. The documentation says the
feature writes the messages of a window "to the same S3 file"; it says nothing
of objects being replaced. The progressive, random and timestamp naming
strategies of the non-grouped mode do give each upload its own key.
*Fix*: the producer counts the completed uploads of each window (under its
lock; an access-ordered map bounded to the last 10000 windows), and a later
upload of a window gets that count after the time range:
{{grouped-messages_20240101_0800_0800-0805-1.txt}}, {{-2}}, ... The first
upload of a window keeps its name. Documentation of the naming in
aws2-s3-streaming.adoc, and a note in the 4.24 upgrade guide (the key of the
second and later objects of a window is new).
*Tests*: new {{AWS2S3StreamUploadTimestampWindowOverwriteTest}} (mocked
{{S3Client}} that keeps completed objects by key, like a bucket).
{{everyMessageOfAWindowIsStoredWhenTheWindowIsUploadedMoreThanOnce}} (the
documentation example, 30 messages in one window) fails on main: {{each upload
of the window must be its own object, but the bucket has
[grouped-messages_20240101_0800_0800-0805.txt] ==> expected: <3> but was:
<1>}}.
{{everyMessageOfAWindowIsStoredWhenTheTimeoutUploadsTheWindowBeforeItEnds}}
(the timeout task runs in the middle of a window) fails on main: {{expected:
<2> but was: <1>}}. {{theFirstUploadOfAWindowKeepsItsName}} passes on main and
with the fix. {{AWS2S3StreamUploadProducerTimestampTest}} is adapted to the new
signature of the name method and checks the suffix. camel-aws2-s3 unit tests:
107 run, 0 failures.
*How it was found*: variant analysis of CAMEL-25781 (pattern: a batch window
completed by a size or count limit or a timer while the window is still open;
what happens to the next item of that window), then the JUnit test.
*Affected*: main, camel-4.23.x, camel-4.22.x and camel-4.18.x (the same naming
in {{processWithTimestampGrouping}}; read, not run). Not run against S3.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
