davsclaus opened a new pull request, #3178:
URL: https://github.com/apache/camel-kamelets/pull/3178

   Part of #3170: `ftp-source`, `ftps-source` and `sftp-source` can move the 
files they read, so that each file is read only once, also across restarts.
   
   ### The problem
   
   Without `delete`, these sources read every file on the server again after a 
restart. The list of files already read (`idempotent`) is kept only in memory.
   
   ### The fix: `move` and `moveFailed`
   
   | Option | What it does |
   |---|---|
   | `move` | Where to move each file on the server once it has been read and 
handed on, such as `.done`. A plain directory name is relative to the directory 
of the file. A path that ends with the file name, such as 
`done/${date:now:yyyyMMdd}/${file:name}`, is relative to `directoryName`. |
   | `moveFailed` | Where to move a file when the next step fails, such as 
`.error`. Without it, a failing file is read again at every poll. |
   
   - **Defaults:** both options are optional. With them unset, the endpoint URI 
is byte-identical to today's.
   - **Descriptions:**
     - Dot-directories such as `.done` are never read, which matters with 
`recursive`.
     - The server user needs the right to create directories and rename files.
     - `move` together with `delete` stops the integration from starting.
   - **Pointers:** the "Good to know" paragraph and the `idempotent` and 
`delete` descriptions point to `move`.
   - **Partial pages:** each has an example.
   
   ### Why not a persistent idempotent repository
   
   A persistent idempotent repository was the first idea, but `move` is better 
here:
   
   - **Less to set up:** `move` needs no Camel knowledge.
   - **Survives redeploys:** the state lives on the server, so it survives pod 
restarts and redeploys. A local file store is lost without a persistent volume.
   - **Not possible from a Kamelet option:** a `FileIdempotentRepository` 
cannot be created only when a file path is given. Without a path it fails to 
start instead of falling back to memory. A bean the user declares cannot set 
`fileStore` from a string either, because Camel has no String to File type 
converter.
   
   A follow-up in `apache/camel`, such as a String to File converter, would 
make an `idempotentRepository` option possible later.
   
   ### Verification
   
   Tested with Camel JBang against `atmoz/sftp`, against 
`delfer/alpine-ftp-server` for FTP, and against the same image with TLS for 
FTPS. Each route ran twice, to simulate a restart.
   
   | Case | Result |
   |---|---|
   | Options unset | The second run read the same files again, as today |
   | `move: .done` and `moveFailed: .error`, with a step that fails for `bad*` 
files | Run 1 moved `a` and `b` to `.done/` and `bad` to `.error/`. Run 2 read 
only the new file `c.txt`. This held on FTP, FTPS and SFTP |
   | `recursive: true` with `.done` | `2026/f.txt` moved to `2026/.done/` and 
was not read again |
   | `move: done/${date:now:yyyyMMdd}/${file:name}` | Moved to 
`upload/done/20261117/h.txt` |
   | `move` with `delete: true` | The integration fails to start, as the 
description says |
   | A failing file without `moveFailed` | Read again at every poll |
   
   `./mvnw clean install` passed twice, the first time with `-DskipTests`.
   
   _Claude Code on behalf of Claus Ibsen_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to