davsclaus opened a new pull request, #3199:
URL: https://github.com/apache/camel-kamelets/pull/3199

   Part of #3170: `ftp-sink`, `ftps-sink` and `sftp-sink` can upload a file 
under a temporary name and rename it once it is complete, and can write a done 
file next to it, so that a reader polling the directory never picks up a 
half-written file.
   
   ### The problem
   
   The sinks upload each file under its real name from the start. In a test, an 
`ftp-source` polling the same directory read `big.bin` at 116 MB while the sink 
was still uploading its 1 GB.
   
   ### The options
   
   | Option | Template mapping | What it does |
   |---|---|---|
   | `tempPrefix` | `tempPrefix: "RAW({{?tempPrefix}})"` | Uploads each file as 
prefix + name, such as `.tmp-orders.csv`, then renames it to `orders.csv`. |
   | `doneFileName` | `doneFileName: "RAW({{?doneFileName}})"` | Once the file 
is complete, writes an empty file next to it, such as `orders.csv.done`. In the 
name, `${file:name}` is the file name and `${file:name.noext}` the name without 
its extension. |
   
   - **Defaults:** both options are optional and deliberately have no 
`default`, as in #3174. When they are unset, the optional placeholders drop 
them, and the endpoint URI is byte-identical to `main`'s.
   - **`RAW()`:** without it, a `+` or `%20` in a value turned into a space. 
With it, `tmp+%20(x)-` and `${file:name}+done 100%(y)` are kept as written, and 
`${file:name}` still works. The value is bound by the operator. The only part 
that comes from the message is the file name, which `tempPrefix` uses without 
its directory.
   - **Why `tempPrefix` and not `tempFileName`:** the component also has 
`tempFileName`, a file-language expression. `tempPrefix` covers the common case 
with plain text. It works with our own sources as they are: `ftp-source`, 
`ftps-source` and `sftp-source` skip names that start with a dot, so `.tmp-` 
needs no setting on the reader. `tempFileName` has a trap: it is relative to 
the directory of the final file, so the natural `${file:name}.tmp` puts the 
subdirectory twice when `CamelFileName` has one. `tempFileName` can be added 
later if someone needs a suffix such as `.part`.
   - **Descriptions:**
     - The "Good to know" paragraph says a file is uploaded under its real 
name, and points to both options.
     - `tempPrefix`: a dot prefix works with Camel readers. A prefix that ends 
with `/`, such as `.inprogress/`, uploads into that subdirectory first; it is 
created unless `autoCreate` is false. With `fileExist` `Override`, the old file 
is deleted when the upload starts, not when it ends. With `Append`, every 
message fails. The user must be allowed to rename files.
     - `doneFileName`: a fixed name is written again after each file. Any other 
`${...}` uploads the file and then fails the message. With `tempPrefix`, the 
done file comes after the rename.
   - **Partial pages:** each has a new "Example with a Temporary Name and a 
Done File".
   
   ### Verification
   
   Tested with Camel JBang 4.24.0-SNAPSHOT:
   - FTP: `delfer/alpine-ftp-server`.
   - FTPS: the same image with explicit TLS (`require_ssl_reuse=NO`, which 
Java's FTPS client needs).
   - SFTP: `atmoz/sftp`.
   
   For the large-file cases, the sink uploaded a 1 GB file while a loop listed 
the server directory every 50 ms. TRACE logging of `GenericFileProducer` showed 
the order of the steps.
   
   | Case | Result |
   |---|---|
   | Options unset, all three Kamelets, `main` and this branch | `camel get 
endpoint` shows identical sink URIs |
   | Unset, 1 GB, FTP / FTPS / SFTP | `big.bin` is visible under its real name 
while it grows (first seen at 0 B, 0 B and 130 KB) |
   | `tempPrefix: .tmp-` and `doneFileName: ${file:name}.done`, 1 GB, FTP / 
FTPS / SFTP | Only `.tmp-big.bin` exists while it grows. `big.bin` is first 
seen at its full 1,000,000,000 bytes, and `big.bin.done` appears at that same 
50 ms sample. Log order: temp write, rename, done file |
   | `tempPrefix` only, FTP | `.tmp-big.bin` while it grows, then `big.bin` 
complete; no done file |
   | `doneFileName: ${file:name.noext}.ready` only, FTP | `big.bin` grows under 
its real name; `big.ready` appears at the first sample where `big.bin` is 
complete |
   | Reader `ftp-source` polling during the upload, unset | Read `big.bin` with 
length 116,391,624 (half-written) |
   | The same with `tempPrefix: .tmp-` | Read `big.bin` once, with length 
1,000,000,000 |
   | `tempPrefix` + `fileExist: Override`, old `big.bin` on the server | The 
old file disappears when `.tmp-big.bin` appears, and the new one appears after 
the rename |
   | `tempPrefix` + `fileExist: Fail`, file exists | The message fails with 
`File already exist`; the old file is kept and no temp file is left |
   | `tempPrefix` + `fileExist: Ignore`, file exists | Old file kept, no done 
file |
   | `tempPrefix` + `fileExist: Append`, FTP and SFTP | The integration starts, 
but every message fails with `You cannot set both fileExist=Append and 
tempPrefix/tempFileName options` |
   | `CamelFileName: 2026/11/orders.csv`, both options | 
`upload/2026/11/.tmp-orders.csv`, renamed to `2026/11/orders.csv`, then 
`2026/11/orders.csv.done` |
   | `tempPrefix: .inprogress/`, FTP / FTPS / SFTP | Uploaded to 
`upload/.inprogress/orders.csv` (directory created), then renamed to 
`upload/orders.csv` |
   | The same with `autoCreate: false`, directory missing | Fails: `550 Failed 
to change directory` |
   | `doneFileName: ready` (fixed), two messages | `ready` is written again 
after each file |
   | `doneFileName: ${date:now:yyyyMMdd}.done` | `orders.csv` is uploaded, then 
the message fails with `Cannot resolve reminder` |
   | `tempPrefix: tmp+%20(x)-`, `doneFileName: ${file:name}+done 100%(y)` | 
Names kept as written: `tmp+%20(x)-orders.csv`, `orders.csv+done 100%(y)` |
   
   `./mvnw clean verify -DskipTests` and then `./mvnw clean verify` passed from 
the root. Before building, the library copies of the three Kamelets were 
refreshed with `./mvnw -pl library/camel-kamelets 
resources:copy-resources@copy-resource-one`. The build regenerated 
`kamelets-index.json`.
   
   _Claude Code on behalf of Claus Ibsen_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to