shashank created CAMEL-25789:
--------------------------------
Summary: camel-aws2-s3 - streaming upload with
namingStrategy=random (or timestamp) names the first file of every producer
keyName alone, so a restart replaces the first file of the previous run
Key: CAMEL-25789
URL: https://issues.apache.org/jira/browse/CAMEL-25789
Project: Camel
Issue Type: Bug
Components: camel-aws2
Reporter: shashank
Assignee: shashank
{{AWS2S3StreamUploadProducer.fileNameToUpload}} (main 7aae27315946, lines
500-550) adds the UUID (namingStrategy {{random}}) or the timestamp
({{timestamp}}) to {{keyName}} only when {{part > 0}}:
{code:java}
case random:
if (part > 0) {
... dynamicKeyName = fileName + "-" + id.toString() + ext;
} else {
... dynamicKeyName = fileName + ext;
}
{code}
{{part}} is the producer's counter of completed uploads ({{UploadState.part}},
line 256), which starts at 0 in every new producer. So the first file of every
producer is named {{keyName}} alone ({{camel.txt}}), and only the later ones
{{camel-<uuid>.txt}}. When the route or the application is started again, the
new producer writes its first file to {{camel.txt}} again, and S3 replaces the
first file of the previous run: its messages ({{batchMessageNumber}}, 10 by
default, or up to {{batchSize}}) are lost at every restart, unless the bucket
is versioned. The same happens between two producers with the same {{keyName}}
(two instances of an application, or two routes). Every exchange completed
normally.
The documentation (aws2-s3-streaming.adoc) says that "with the random strategy
a UUID will be added after keyName", and that restarting from the last part
"won't be needed when using the random naming strategy" (the
{{restartingPolicy}} {{lastPart}} only works with {{progressive}}, whose
{{override}} default replaces the earlier files by design). So a user who picks
{{random}} to keep the files of every run loses one file per restart.
{{AWSS3NamingStrategyTest}} pins the name of the first file ({{testFile.txt}}
for part 0), but nothing explains why the first file should have no random part.
*Fix*: with {{random}} and {{timestamp}} every upload gets the UUID or the
timestamp, the first one too. {{progressive}} is unchanged. Documentation of
the three strategies in aws2-s3-streaming.adoc, and a note in the 4.24 upgrade
guide (the first file has a new name).
*Tests*: new {{AWS2S3StreamUploadRandomNamingRestartTest}} (mocked {{S3Client}}
that keeps the completed objects by key, like a bucket).
{{aProducerStartedAgainDoesNotReplaceTheObjectsOfTheFirstOne}} (15 messages,
stop, a new producer, 15 messages, stop) fails on main: {{every upload must be
its own object, but the bucket has [camel-7c06...txt, camel-91f0...txt,
camel.txt] ==> expected: <4> but was: <3>}}.
{{everyObjectHasTheKeyNameAndARandomPart}} fails on main: {{camel.txt ==>
expected: <true> but was: <false>}}. {{AWSS3NamingStrategyTest}} now expects
the random part and the timestamp for part 0. camel-aws2-s3 unit tests: 106
run, 0 failures.
*How it was found*: variant analysis of CAMEL-25759 (pattern: a name that has
to be unique across starts is built from per-start state, so a restart or a
second instance reuses it), found while working on the timestamp grouping of
the same producer. Then the JUnit test.
*Affected*: main, camel-4.23.x, camel-4.22.x, camel-4.18.x and camel-4.14.x
(the same {{part > 0}} condition; read, not run). The behaviour dates from
CAMEL-16185. Not run against S3.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
