davsclaus opened a new pull request, #3199:
URL: https://github.com/apache/camel-kamelets/pull/3199
Part of #3170: `ftp-sink`, `ftps-sink` and `sftp-sink` can upload a file
under a temporary name and rename it once it is complete, and can write a done
file next to it, so that a reader polling the directory never picks up a
half-written file.
### The problem
The sinks upload each file under its real name from the start. In a test, an
`ftp-source` polling the same directory read `big.bin` at 116 MB while the sink
was still uploading its 1 GB.
### The options
| Option | Template mapping | What it does |
|---|---|---|
| `tempPrefix` | `tempPrefix: "RAW({{?tempPrefix}})"` | Uploads each file as
prefix + name, such as `.tmp-orders.csv`, then renames it to `orders.csv`. |
| `doneFileName` | `doneFileName: "RAW({{?doneFileName}})"` | Once the file
is complete, writes an empty file next to it, such as `orders.csv.done`. In the
name, `${file:name}` is the file name and `${file:name.noext}` the name without
its extension. |
- **Defaults:** both options are optional and deliberately have no
`default`, as in #3174. When they are unset, the optional placeholders drop
them, and the endpoint URI is byte-identical to `main`'s.
- **`RAW()`:** without it, a `+` or `%20` in a value turned into a space.
With it, `tmp+%20(x)-` and `${file:name}+done 100%(y)` are kept as written, and
`${file:name}` still works. The value is bound by the operator. The only part
that comes from the message is the file name, which `tempPrefix` uses without
its directory.
- **Why `tempPrefix` and not `tempFileName`:** the component also has
`tempFileName`, a file-language expression. `tempPrefix` covers the common case
with plain text. It works with our own sources as they are: `ftp-source`,
`ftps-source` and `sftp-source` skip names that start with a dot, so `.tmp-`
needs no setting on the reader. `tempFileName` has a trap: it is relative to
the directory of the final file, so the natural `${file:name}.tmp` puts the
subdirectory twice when `CamelFileName` has one. `tempFileName` can be added
later if someone needs a suffix such as `.part`.
- **Descriptions:**
- The "Good to know" paragraph says a file is uploaded under its real
name, and points to both options.
- `tempPrefix`: a dot prefix works with Camel readers. A prefix that ends
with `/`, such as `.inprogress/`, uploads into that subdirectory first; it is
created unless `autoCreate` is false. With `fileExist` `Override`, the old file
is deleted when the upload starts, not when it ends. With `Append`, every
message fails. The user must be allowed to rename files.
- `doneFileName`: a fixed name is written again after each file. Any other
`${...}` uploads the file and then fails the message. With `tempPrefix`, the
done file comes after the rename.
- **Partial pages:** each has a new "Example with a Temporary Name and a
Done File".
### Verification
Tested with Camel JBang 4.24.0-SNAPSHOT:
- FTP: `delfer/alpine-ftp-server`.
- FTPS: the same image with explicit TLS (`require_ssl_reuse=NO`, which
Java's FTPS client needs).
- SFTP: `atmoz/sftp`.
For the large-file cases, the sink uploaded a 1 GB file while a loop listed
the server directory every 50 ms. TRACE logging of `GenericFileProducer` showed
the order of the steps.
| Case | Result |
|---|---|
| Options unset, all three Kamelets, `main` and this branch | `camel get
endpoint` shows identical sink URIs |
| Unset, 1 GB, FTP / FTPS / SFTP | `big.bin` is visible under its real name
while it grows (first seen at 0 B, 0 B and 130 KB) |
| `tempPrefix: .tmp-` and `doneFileName: ${file:name}.done`, 1 GB, FTP /
FTPS / SFTP | Only `.tmp-big.bin` exists while it grows. `big.bin` is first
seen at its full 1,000,000,000 bytes, and `big.bin.done` appears at that same
50 ms sample. Log order: temp write, rename, done file |
| `tempPrefix` only, FTP | `.tmp-big.bin` while it grows, then `big.bin`
complete; no done file |
| `doneFileName: ${file:name.noext}.ready` only, FTP | `big.bin` grows under
its real name; `big.ready` appears at the first sample where `big.bin` is
complete |
| Reader `ftp-source` polling during the upload, unset | Read `big.bin` with
length 116,391,624 (half-written) |
| The same with `tempPrefix: .tmp-` | Read `big.bin` once, with length
1,000,000,000 |
| `tempPrefix` + `fileExist: Override`, old `big.bin` on the server | The
old file disappears when `.tmp-big.bin` appears, and the new one appears after
the rename |
| `tempPrefix` + `fileExist: Fail`, file exists | The message fails with
`File already exist`; the old file is kept and no temp file is left |
| `tempPrefix` + `fileExist: Ignore`, file exists | Old file kept, no done
file |
| `tempPrefix` + `fileExist: Append`, FTP and SFTP | The integration starts,
but every message fails with `You cannot set both fileExist=Append and
tempPrefix/tempFileName options` |
| `CamelFileName: 2026/11/orders.csv`, both options |
`upload/2026/11/.tmp-orders.csv`, renamed to `2026/11/orders.csv`, then
`2026/11/orders.csv.done` |
| `tempPrefix: .inprogress/`, FTP / FTPS / SFTP | Uploaded to
`upload/.inprogress/orders.csv` (directory created), then renamed to
`upload/orders.csv` |
| The same with `autoCreate: false`, directory missing | Fails: `550 Failed
to change directory` |
| `doneFileName: ready` (fixed), two messages | `ready` is written again
after each file |
| `doneFileName: ${date:now:yyyyMMdd}.done` | `orders.csv` is uploaded, then
the message fails with `Cannot resolve reminder` |
| `tempPrefix: tmp+%20(x)-`, `doneFileName: ${file:name}+done 100%(y)` |
Names kept as written: `tmp+%20(x)-orders.csv`, `orders.csv+done 100%(y)` |
`./mvnw clean verify -DskipTests` and then `./mvnw clean verify` passed from
the root. Before building, the library copies of the three Kamelets were
refreshed with `./mvnw -pl library/camel-kamelets
resources:copy-resources@copy-resource-one`. The build regenerated
`kamelets-index.json`.
_Claude Code on behalf of Claus Ibsen_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]