This is an automated email from the ASF dual-hosted git repository.
oscerd pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel-kamelets.git
The following commit(s) were added to refs/heads/main by this push:
new de849a18f chore: the FTP and SFTP sources can move or remember the
files they read, to read each only once across restarts (#3170) (#3178)
de849a18f is described below
commit de849a18fa30e785b01aa366d24e604db8490c18
Author: Claus Ibsen <[email protected]>
AuthorDate: Wed Nov 18 09:57:52 2026 +0100
chore: the FTP and SFTP sources can move or remember the files they read,
to read each only once across restarts (#3170) (#3178)
* chore: the FTP and SFTP sources can move the files they read, to read
each only once across restarts (#3170)
ftp-source, ftps-source and sftp-source get move and moveFailed. With move
set, each file read is moved on the server, so it is not read again after a
restart, which the in-memory idempotent list cannot prevent. moveFailed
takes a
failing file out of the way. Unset, the endpoint URI is the same as before.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* chore: the FTP and SFTP sources take an idempotentRepository (#3170)
ftp-source, ftps-source and sftp-source get idempotentRepository: the name
of
a bean, such as a FileIdempotentRepository, that keeps the list of the files
read, so that files left on the server are not read again after a restart.
Mapped as #bean:{{?idempotentRepository}}, so the in-memory list stays the
default when it is not set. The partial pages show a YAML bean example.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
.../ROOT/partials/ftp-source-description.adoc | 55 ++++++++++++++++++++-
.../ROOT/partials/ftps-source-description.adoc | 57 +++++++++++++++++++++-
.../ROOT/partials/sftp-source-description.adoc | 55 ++++++++++++++++++++-
kamelets/ftp-source.kamelet.yaml | 24 +++++++--
kamelets/ftps-source.kamelet.yaml | 24 +++++++--
kamelets/sftp-source.kamelet.yaml | 24 +++++++--
.../main/resources/META-INF/kamelets-index.json | 6 +--
.../resources/kamelets/ftp-source.kamelet.yaml | 24 +++++++--
.../resources/kamelets/ftps-source.kamelet.yaml | 24 +++++++--
.../resources/kamelets/sftp-source.kamelet.yaml | 24 +++++++--
10 files changed, 292 insertions(+), 25 deletions(-)
diff --git a/docs/modules/ROOT/partials/ftp-source-description.adoc
b/docs/modules/ROOT/partials/ftp-source-description.adoc
index 3ffe27f5f..8d7d66bad 100644
--- a/docs/modules/ROOT/partials/ftp-source-description.adoc
+++ b/docs/modules/ROOT/partials/ftp-source-description.adoc
@@ -18,7 +18,9 @@ The FTP Source Kamelet supports the following configurations:
- **Idempotent**: Skip already-processed files (default: true)
- **Binary**: Use binary transfer mode instead of ASCII (default: false)
- **Auto Create**: Automatically create the starting directory if it doesn't
exist (default: true)
-- **Delete**: Delete files after successful processing (default: false). When
false, files are left on the server; with **Idempotent**, they are skipped
while the integration runs but read again after a restart
+- **Delete**: Delete files after successful processing (default: false). When
false, and **Move** is not set, files are left on the server; with
**Idempotent**, they are skipped while the integration runs but read again
after a restart
+- **Move**: Where to move each file on the server after it has been read, such
as `.done`, so that it is not read again, also after a restart (not set by
default). It cannot be combined with **Delete**
+- **Move Failed**: Where to move a file on the server when it could not be
handed on, such as `.error` (not set by default). When not set, a file that
fails is read again at every poll
=== Output Format
@@ -65,6 +67,57 @@ The Kamelet outputs file content as an `InputStream` and
sets headers with file
uri: "kamelet:log-sink"
----
+=== Example with Files Moved After Reading
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:ftp-source"
+ parameters:
+ connectionHost: "ftp.example.com"
+ username: "ftpuser"
+ password: "ftppass"
+ directoryName: "/incoming"
+ move: ".done"
+ moveFailed: ".error"
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+Each file is moved to the `.done` directory next to it once it has been read,
and to `.error` when it could not be handed on, so a restart does not read the
same files again while they are kept on the server. Directories whose names
start with a dot are never read, also when **Recursive** is true. The user
needs the right to create directories and rename files on the server.
+
+=== Example with Files Remembered Across Restarts
+
+To keep the files on the server and still read each one only once, also across
restarts, keep the list of the files
+read in a file. Declare a `FileIdempotentRepository` bean that writes the list
to a file, and give its name in
+**idempotentRepository**:
+
+[source,yaml,subs='+attributes,macros']
+----
+- beans:
+ - name: readFiles
+ type:
"#class:org.apache.camel.support.processor.idempotent.FileIdempotentRepository"
+ properties:
+ fileStore: "#class:java.io.File('/var/data/read-files.dat')"
+- route:
+ from:
+ uri: "kamelet:ftp-source"
+ parameters:
+ connectionHost: "ftp.example.com"
+ username: "ftpuser"
+ password: "ftppass"
+ directoryName: "/incoming"
+ idempotentRepository: "readFiles"
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+The file must be on storage that survives a restart of the integration, such
as a persistent volume. A file that is
+put on the server again under the same name is not read again.
+
=== Example with File Deletion
[source,yaml,subs='+attributes,macros']
diff --git a/docs/modules/ROOT/partials/ftps-source-description.adoc
b/docs/modules/ROOT/partials/ftps-source-description.adoc
index 347de192b..7cc128259 100644
--- a/docs/modules/ROOT/partials/ftps-source-description.adoc
+++ b/docs/modules/ROOT/partials/ftps-source-description.adoc
@@ -13,13 +13,15 @@ The FTPS Source Kamelet supports the following
configurations:
- **Username**: Username for FTPS authentication (required)
- **Password**: Password for FTPS authentication (required)
- **Directory Name**: The starting directory path on the FTPS server (required)
-- Files are left on the server after they are read unless **Delete** is true;
with **Idempotent**, they are skipped while the integration runs but read again
after a restart
+- Files are left on the server after they are read unless **Delete** is true
or **Move** is set; with **Idempotent**, they are skipped while the integration
runs but read again after a restart
- **Passive Mode**: Use passive mode for FTPS connections (default: false)
- **Recursive**: Process files in subdirectories (default: false)
- **Idempotent**: Skip already-processed files (default: true)
- **Binary**: Use binary transfer mode instead of ASCII (default: false)
- **Auto Create**: Automatically create the starting directory if it doesn't
exist (default: true)
- **Delete**: Delete files after successful processing (default: false)
+- **Move**: Where to move each file on the server after it has been read, such
as `.done`, so that it is not read again, also after a restart (not set by
default). It cannot be combined with **Delete**
+- **Move Failed**: Where to move a file on the server when it could not be
handed on, such as `.error` (not set by default). When not set, a file that
fails is read again at every poll
=== Output Format
@@ -70,4 +72,55 @@ FTPS encrypts the connection of standard FTP with TLS. The
Kamelet uses explicit
uri: "kamelet:log-sink"
----
-This example enables passive mode, binary transfer, and recursive directory
processing. The Kamelet uses explicit FTPS: it connects to the normal FTP port
and then switches to TLS. Servers that only accept implicit FTPS (usually on
port 990) are not supported.
\ No newline at end of file
+This example enables passive mode, binary transfer, and recursive directory
processing. The Kamelet uses explicit FTPS: it connects to the normal FTP port
and then switches to TLS. Servers that only accept implicit FTPS (usually on
port 990) are not supported.
+
+=== Example with Files Moved After Reading
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:ftps-source"
+ parameters:
+ connectionHost: "ftps.example.com"
+ username: "ftpsuser"
+ password: "ftpspass"
+ directoryName: "/incoming"
+ move: ".done"
+ moveFailed: ".error"
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+Each file is moved to the `.done` directory next to it once it has been read,
and to `.error` when it could not be handed on, so a restart does not read the
same files again while they are kept on the server. Directories whose names
start with a dot are never read, also when **Recursive** is true. The user
needs the right to create directories and rename files on the server.
+
+=== Example with Files Remembered Across Restarts
+
+To keep the files on the server and still read each one only once, also across
restarts, keep the list of the files
+read in a file. Declare a `FileIdempotentRepository` bean that writes the list
to a file, and give its name in
+**idempotentRepository**:
+
+[source,yaml,subs='+attributes,macros']
+----
+- beans:
+ - name: readFiles
+ type:
"#class:org.apache.camel.support.processor.idempotent.FileIdempotentRepository"
+ properties:
+ fileStore: "#class:java.io.File('/var/data/read-files.dat')"
+- route:
+ from:
+ uri: "kamelet:ftps-source"
+ parameters:
+ connectionHost: "ftps.example.com"
+ username: "ftpsuser"
+ password: "ftpspass"
+ directoryName: "/incoming"
+ idempotentRepository: "readFiles"
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+The file must be on storage that survives a restart of the integration, such
as a persistent volume. A file that is
+put on the server again under the same name is not read again.
diff --git a/docs/modules/ROOT/partials/sftp-source-description.adoc
b/docs/modules/ROOT/partials/sftp-source-description.adoc
index 194873ccc..11cbdf038 100644
--- a/docs/modules/ROOT/partials/sftp-source-description.adoc
+++ b/docs/modules/ROOT/partials/sftp-source-description.adoc
@@ -17,7 +17,9 @@ The SFTP Source Kamelet supports the following configurations:
- **Idempotent**: Skip already-processed files (default: true)
- **Binary**: Use binary transfer mode instead of ASCII (default: false)
- **Auto Create**: Automatically create the starting directory if it doesn't
exist (default: true)
-- **Delete**: Delete files after successful processing (default: false). When
false, files are left on the server; with **Idempotent**, they are skipped
while the integration runs but read again after a restart
+- **Delete**: Delete files after successful processing (default: false). When
false, and **Move** is not set, files are left on the server; with
**Idempotent**, they are skipped while the integration runs but read again
after a restart
+- **Move**: Where to move each file on the server after it has been read, such
as `.done`, so that it is not read again, also after a restart (not set by
default). It cannot be combined with **Delete**
+- **Move Failed**: Where to move a file on the server when it could not be
handed on, such as `.error` (not set by default). When not set, a file that
fails is read again at every poll
=== Output Format
@@ -66,6 +68,57 @@ SFTP provides enhanced security over standard FTP by using
SSH for authenticatio
uri: "kamelet:log-sink"
----
+=== Example with Files Moved After Reading
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:sftp-source"
+ parameters:
+ connectionHost: "sftp.example.com"
+ username: "sftpuser"
+ password: "sftppass"
+ directoryName: "/incoming"
+ move: ".done"
+ moveFailed: ".error"
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+Each file is moved to the `.done` directory next to it once it has been read,
and to `.error` when it could not be handed on, so a restart does not read the
same files again while they are kept on the server. Directories whose names
start with a dot are never read, also when **Recursive** is true. The user
needs the right to create directories and rename files on the server.
+
+=== Example with Files Remembered Across Restarts
+
+To keep the files on the server and still read each one only once, also across
restarts, keep the list of the files
+read in a file. Declare a `FileIdempotentRepository` bean that writes the list
to a file, and give its name in
+**idempotentRepository**:
+
+[source,yaml,subs='+attributes,macros']
+----
+- beans:
+ - name: readFiles
+ type:
"#class:org.apache.camel.support.processor.idempotent.FileIdempotentRepository"
+ properties:
+ fileStore: "#class:java.io.File('/var/data/read-files.dat')"
+- route:
+ from:
+ uri: "kamelet:sftp-source"
+ parameters:
+ connectionHost: "sftp.example.com"
+ username: "sftpuser"
+ password: "sftppass"
+ directoryName: "/incoming"
+ idempotentRepository: "readFiles"
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+The file must be on storage that survives a restart of the integration, such
as a persistent volume. A file that is
+put on the server again under the same name is not read again.
+
=== Example with File Deletion
[source,yaml,subs='+attributes,macros']
diff --git a/kamelets/ftp-source.kamelet.yaml b/kamelets/ftp-source.kamelet.yaml
index 362155ac7..0c77a7c79 100644
--- a/kamelets/ftp-source.kamelet.yaml
+++ b/kamelets/ftp-source.kamelet.yaml
@@ -37,7 +37,7 @@ spec:
Output: the body is the content of the file, as a stream. The `file` and
`ce-file` headers hold the name of the file, with its path below
`directoryName` when `recursive` is true, such as `2026/orders.csv`.
`CamelFileName` holds the same name, and `CamelFileLength` the size of the file
in bytes.
- Good to know: files are not deleted or moved after they are read, unless
`delete` is true. A file is read only once while the integration runs
(`idempotent`), but the files left on the server are all read again after a
restart; set `delete` to true to read each file only once. The connection and
the login are not encrypted. Files are transferred in ASCII (text) mode unless
`binary` is true.
+ Good to know: files are not deleted or moved after they are read, unless
`delete` is true or `move` is set. A file is read only once while the
integration runs (`idempotent`), but the files left on the server are all read
again after a restart; to read each file only once, also across restarts, set
`move` to a directory such as `.done`, set `delete` to true, or keep the list
of the files read in a file with `idempotentRepository`. The connection and the
login are not encrypted. Fil [...]
required:
- connectionHost
- connectionPort
@@ -86,9 +86,14 @@ spec:
default: false
idempotent:
title: Skip Files Already Read
- description: If true, a file is read only once while the integration
runs, even when it stays on the server. The list of the files read is kept in
memory, so after a restart the files still on the server are read again. If
false, and `delete` is false, the same files are read again on every poll.
+ description: If true, a file is read only once while the integration
runs, even when it stays on the server. The list of the files read is kept in
memory, so after a restart the files still on the server are read again, unless
`move` or `delete` takes them away, or `idempotentRepository` keeps the list
elsewhere. If false, and neither `delete` nor `move` is set, the same files are
read again on every poll.
type: boolean
default: true
+ idempotentRepository:
+ title: Files Read Store
+ description: The name of a bean that keeps the list of the files read,
so that the files left on the server are not read again after a restart, such
as a `FileIdempotentRepository` that keeps the list in a file. Declare the bean
in the integration, for example under `beans` in YAML. Only used when
`idempotent` is true. When it is not set, the list is kept in memory.
+ type: string
+ example: readFiles
binary:
title: Binary Mode
description: If true, files are transferred in binary mode, byte for
byte. When false, they are transferred in ASCII mode, which is meant for text.
Set it to true for files that are not text, such as images, archives or PDF
files.
@@ -101,9 +106,19 @@ spec:
default: true
delete:
title: Delete
- description: If true, each file is deleted from the server once it has
been read and handed on successfully. Use it to read each file only once, also
across restarts.
+ description: If true, each file is deleted from the server once it has
been read and handed on successfully. Use it to read each file only once, also
across restarts. To keep the files, set `move` instead.
type: boolean
default: false
+ move:
+ title: Move After Read
+ description: Where to move each file on the server once it has been
read and handed on successfully, so that it is not read again, also after a
restart. A directory name, such as `.done`, is relative to the directory of the
file. A path that ends with the name of the file, such as
`done/$\{date:now:yyyyMMdd}/$\{file:name}`, is relative to `directoryName`,
where `$\{file:name}` is the name of the file and `$\{date:now:yyyyMMdd}`
today's date. A directory whose name starts with a d [...]
+ type: string
+ example: ".done"
+ moveFailed:
+ title: Move Failed Files To
+ description: Where to move a file on the server when it could not be
handed on, for example because the next step failed, so that it is not read
over and over. The same format as `move`, such as `.error`. When it is not set,
a file that fails stays where it is and is read again at every poll. It can be
set together with `delete` or `move`.
+ type: string
+ example: ".error"
dataTypes:
out:
headers:
@@ -137,6 +152,9 @@ spec:
binary: "{{binary}}"
delete: "{{delete}}"
idempotent: "{{idempotent}}"
+ idempotentRepository: "#bean:{{?idempotentRepository}}"
+ move: "{{?move}}"
+ moveFailed: "{{?moveFailed}}"
passiveMode: "{{passiveMode}}"
password: "RAW({{password}})"
recursive: "{{recursive}}"
diff --git a/kamelets/ftps-source.kamelet.yaml
b/kamelets/ftps-source.kamelet.yaml
index 7e733e423..9564bbcf9 100644
--- a/kamelets/ftps-source.kamelet.yaml
+++ b/kamelets/ftps-source.kamelet.yaml
@@ -37,7 +37,7 @@ spec:
Output: the body is the content of the file, as a stream. The `file` and
`ce-file` headers hold the name of the file, with its path below
`directoryName` when `recursive` is true, such as `2026/orders.csv`.
`CamelFileName` holds the same name, and `CamelFileLength` the size of the file
in bytes.
- Good to know: files are not deleted or moved after they are read, unless
`delete` is true. A file is read only once while the integration runs
(`idempotent`), but the files left on the server are all read again after a
restart; set `delete` to true to read each file only once. It uses explicit
TLS: it connects to the normal FTP port, usually 21, and then switches to TLS.
A server that only accepts implicit TLS, usually on port 990, is not supported.
Files are transferred in ASCII ( [...]
+ Good to know: files are not deleted or moved after they are read, unless
`delete` is true or `move` is set. A file is read only once while the
integration runs (`idempotent`), but the files left on the server are all read
again after a restart; to read each file only once, also across restarts, set
`move` to a directory such as `.done`, set `delete` to true, or keep the list
of the files read in a file with `idempotentRepository`. It uses explicit TLS:
it connects to the normal FTP [...]
required:
- connectionHost
- connectionPort
@@ -86,9 +86,14 @@ spec:
default: false
idempotent:
title: Skip Files Already Read
- description: If true, a file is read only once while the integration
runs, even when it stays on the server. The list of the files read is kept in
memory, so after a restart the files still on the server are read again. If
false, and `delete` is false, the same files are read again on every poll.
+ description: If true, a file is read only once while the integration
runs, even when it stays on the server. The list of the files read is kept in
memory, so after a restart the files still on the server are read again, unless
`move` or `delete` takes them away, or `idempotentRepository` keeps the list
elsewhere. If false, and neither `delete` nor `move` is set, the same files are
read again on every poll.
type: boolean
default: true
+ idempotentRepository:
+ title: Files Read Store
+ description: The name of a bean that keeps the list of the files read,
so that the files left on the server are not read again after a restart, such
as a `FileIdempotentRepository` that keeps the list in a file. Declare the bean
in the integration, for example under `beans` in YAML. Only used when
`idempotent` is true. When it is not set, the list is kept in memory.
+ type: string
+ example: readFiles
binary:
title: Binary Mode
description: If true, files are transferred in binary mode, byte for
byte. When false, they are transferred in ASCII mode, which is meant for text.
Set it to true for files that are not text, such as images, archives or PDF
files.
@@ -101,9 +106,19 @@ spec:
default: true
delete:
title: Delete
- description: If true, each file is deleted from the server once it has
been read and handed on successfully. Use it to read each file only once, also
across restarts.
+ description: If true, each file is deleted from the server once it has
been read and handed on successfully. Use it to read each file only once, also
across restarts. To keep the files, set `move` instead.
type: boolean
default: false
+ move:
+ title: Move After Read
+ description: Where to move each file on the server once it has been
read and handed on successfully, so that it is not read again, also after a
restart. A directory name, such as `.done`, is relative to the directory of the
file. A path that ends with the name of the file, such as
`done/$\{date:now:yyyyMMdd}/$\{file:name}`, is relative to `directoryName`,
where `$\{file:name}` is the name of the file and `$\{date:now:yyyyMMdd}`
today's date. A directory whose name starts with a d [...]
+ type: string
+ example: ".done"
+ moveFailed:
+ title: Move Failed Files To
+ description: Where to move a file on the server when it could not be
handed on, for example because the next step failed, so that it is not read
over and over. The same format as `move`, such as `.error`. When it is not set,
a file that fails stays where it is and is read again at every poll. It can be
set together with `delete` or `move`.
+ type: string
+ example: ".error"
dataTypes:
out:
headers:
@@ -137,6 +152,9 @@ spec:
binary: "{{binary}}"
delete: "{{delete}}"
idempotent: "{{idempotent}}"
+ idempotentRepository: "#bean:{{?idempotentRepository}}"
+ move: "{{?move}}"
+ moveFailed: "{{?moveFailed}}"
passiveMode: "{{passiveMode}}"
password: "RAW({{password}})"
recursive: "{{recursive}}"
diff --git a/kamelets/sftp-source.kamelet.yaml
b/kamelets/sftp-source.kamelet.yaml
index a21c2f781..f4697f038 100644
--- a/kamelets/sftp-source.kamelet.yaml
+++ b/kamelets/sftp-source.kamelet.yaml
@@ -37,7 +37,7 @@ spec:
Output: the body is the content of the file, as bytes. The `file` and
`ce-file` headers hold the name of the file, with its path below
`directoryName` when `recursive` is true, such as `2026/orders.csv`.
`CamelFileName` holds the same name, and `CamelFileLength` the size of the file
in bytes.
- Good to know: files are not deleted or moved after they are read, unless
`delete` is true. A file is read only once while the integration runs
(`idempotent`), but the files left on the server are all read again after a
restart; set `delete` to true to read each file only once. Log in with
`username` and either `password` or a private key. The key of the server must
be in a known_hosts file, `knownHostsUri` or else `~/.ssh/known_hosts` of the
user that runs the integration, or the c [...]
+ Good to know: files are not deleted or moved after they are read, unless
`delete` is true or `move` is set. A file is read only once while the
integration runs (`idempotent`), but the files left on the server are all read
again after a restart; to read each file only once, also across restarts, set
`move` to a directory such as `.done`, set `delete` to true, or keep the list
of the files read in a file with `idempotentRepository`. Log in with `username`
and either `password` or a p [...]
required:
- connectionHost
- connectionPort
@@ -84,9 +84,14 @@ spec:
default: false
idempotent:
title: Skip Files Already Read
- description: If true, a file is read only once while the integration
runs, even when it stays on the server. The list of the files read is kept in
memory, so after a restart the files still on the server are read again. If
false, and `delete` is false, the same files are read again on every poll.
+ description: If true, a file is read only once while the integration
runs, even when it stays on the server. The list of the files read is kept in
memory, so after a restart the files still on the server are read again, unless
`move` or `delete` takes them away, or `idempotentRepository` keeps the list
elsewhere. If false, and neither `delete` nor `move` is set, the same files are
read again on every poll.
type: boolean
default: true
+ idempotentRepository:
+ title: Files Read Store
+ description: The name of a bean that keeps the list of the files read,
so that the files left on the server are not read again after a restart, such
as a `FileIdempotentRepository` that keeps the list in a file. Declare the bean
in the integration, for example under `beans` in YAML. Only used when
`idempotent` is true. When it is not set, the list is kept in memory.
+ type: string
+ example: readFiles
ignoreFileNotFoundOrPermissionError:
title: Ignore Access Errors
description: If true, a directory or a file that does not exist, or
that `username` may not read, is skipped. If false, it causes an error.
@@ -139,9 +144,19 @@ spec:
default: true
delete:
title: Delete
- description: If true, each file is deleted from the server once it has
been read and handed on successfully. Use it to read each file only once, also
across restarts.
+ description: If true, each file is deleted from the server once it has
been read and handed on successfully. Use it to read each file only once, also
across restarts. To keep the files, set `move` instead.
type: boolean
default: false
+ move:
+ title: Move After Read
+ description: Where to move each file on the server once it has been
read and handed on successfully, so that it is not read again, also after a
restart. A directory name, such as `.done`, is relative to the directory of the
file. A path that ends with the name of the file, such as
`done/$\{date:now:yyyyMMdd}/$\{file:name}`, is relative to `directoryName`,
where `$\{file:name}` is the name of the file and `$\{date:now:yyyyMMdd}`
today's date. A directory whose name starts with a d [...]
+ type: string
+ example: ".done"
+ moveFailed:
+ title: Move Failed Files To
+ description: Where to move a file on the server when it could not be
handed on, for example because the next step failed, so that it is not read
over and over. The same format as `move`, such as `.error`. When it is not set,
a file that fails stays where it is and is read again at every poll. It can be
set together with `delete` or `move`.
+ type: string
+ example: ".error"
dataTypes:
out:
headers:
@@ -175,8 +190,11 @@ spec:
binary: "{{binary}}"
delete: "{{delete}}"
idempotent: "{{idempotent}}"
+ idempotentRepository: "#bean:{{?idempotentRepository}}"
ignoreFileNotFoundOrPermissionError:
"{{ignoreFileNotFoundOrPermissionError}}"
knownHostsUri: "RAW({{?knownHostsUri}})"
+ move: "{{?move}}"
+ moveFailed: "{{?moveFailed}}"
passiveMode: "{{passiveMode}}"
password: "RAW({{?password}})"
privateKeyFile: "RAW({{?privateKeyFile}})"
diff --git
a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
index 9747e2ab3..8aa0b94ea 100644
--- a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
+++ b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
@@ -795,7 +795,7 @@
"supportLevel" : "Stable",
"title" : "FTP Source",
"summary" : "Reads the files in a directory on an FTP server, one message
per file, and by default leaves them on the server.",
- "description" : "Reads the files in a directory on an FTP server, one
message per file, and by default leaves them on the server.\n\nUse it to
collect the files that a partner or a legacy system drops on an FTP server,
such as orders, invoices or exports. For an encrypted connection use
`ftps-source`, and for a server you reach with SSH use
`sftp-source`.\n\nOutput: the body is the content of the file, as a stream. The
`file` and `ce-file` headers hold the name of the file, with its [...]
+ "description" : "Reads the files in a directory on an FTP server, one
message per file, and by default leaves them on the server.\n\nUse it to
collect the files that a partner or a legacy system drops on an FTP server,
such as orders, invoices or exports. For an encrypted connection use
`ftps-source`, and for a server you reach with SSH use
`sftp-source`.\n\nOutput: the body is the content of the file, as a stream. The
`file` and `ce-file` headers hold the name of the file, with its [...]
"group" : "FTP",
"namespace" : "File",
"provider" : "Apache Software Foundation"
@@ -815,7 +815,7 @@
"supportLevel" : "Stable",
"title" : "FTPS Source",
"summary" : "Reads the files in a directory on an FTPS server (FTP with
TLS encryption), one message per file, and by default leaves them on the
server.",
- "description" : "Reads the files in a directory on an FTPS server (FTP
with TLS encryption), one message per file, and by default leaves them on the
server.\n\nUse it to collect the files that a partner or a legacy system drops
on an FTPS server, such as orders, invoices or exports. For a server without
TLS use `ftp-source`, and for a server you reach with SSH use
`sftp-source`.\n\nOutput: the body is the content of the file, as a stream. The
`file` and `ce-file` headers hold the nam [...]
+ "description" : "Reads the files in a directory on an FTPS server (FTP
with TLS encryption), one message per file, and by default leaves them on the
server.\n\nUse it to collect the files that a partner or a legacy system drops
on an FTPS server, such as orders, invoices or exports. For a server without
TLS use `ftp-source`, and for a server you reach with SSH use
`sftp-source`.\n\nOutput: the body is the content of the file, as a stream. The
`file` and `ce-file` headers hold the nam [...]
"group" : "FTP",
"namespace" : "File",
"provider" : "Apache Software Foundation"
@@ -2035,7 +2035,7 @@
"supportLevel" : "Stable",
"title" : "SFTP Source",
"summary" : "Reads the files in a directory on an SFTP server, over an
encrypted SSH connection, one message per file, and by default leaves them on
the server.",
- "description" : "Reads the files in a directory on an SFTP server, over an
encrypted SSH connection, one message per file, and by default leaves them on
the server.\n\nUse it to collect the files that a partner or another system
drops on a server you reach with SSH, such as orders, invoices or exports. For
an FTP server use `ftp-source` or `ftps-source`.\n\nOutput: the body is the
content of the file, as bytes. The `file` and `ce-file` headers hold the name
of the file, with its path [...]
+ "description" : "Reads the files in a directory on an SFTP server, over an
encrypted SSH connection, one message per file, and by default leaves them on
the server.\n\nUse it to collect the files that a partner or another system
drops on a server you reach with SSH, such as orders, invoices or exports. For
an FTP server use `ftp-source` or `ftps-source`.\n\nOutput: the body is the
content of the file, as bytes. The `file` and `ce-file` headers hold the name
of the file, with its path [...]
"group" : "FTP",
"namespace" : "File",
"provider" : "Apache Software Foundation"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/ftp-source.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/ftp-source.kamelet.yaml
index 362155ac7..0c77a7c79 100644
--- a/library/camel-kamelets/src/main/resources/kamelets/ftp-source.kamelet.yaml
+++ b/library/camel-kamelets/src/main/resources/kamelets/ftp-source.kamelet.yaml
@@ -37,7 +37,7 @@ spec:
Output: the body is the content of the file, as a stream. The `file` and
`ce-file` headers hold the name of the file, with its path below
`directoryName` when `recursive` is true, such as `2026/orders.csv`.
`CamelFileName` holds the same name, and `CamelFileLength` the size of the file
in bytes.
- Good to know: files are not deleted or moved after they are read, unless
`delete` is true. A file is read only once while the integration runs
(`idempotent`), but the files left on the server are all read again after a
restart; set `delete` to true to read each file only once. The connection and
the login are not encrypted. Files are transferred in ASCII (text) mode unless
`binary` is true.
+ Good to know: files are not deleted or moved after they are read, unless
`delete` is true or `move` is set. A file is read only once while the
integration runs (`idempotent`), but the files left on the server are all read
again after a restart; to read each file only once, also across restarts, set
`move` to a directory such as `.done`, set `delete` to true, or keep the list
of the files read in a file with `idempotentRepository`. The connection and the
login are not encrypted. Fil [...]
required:
- connectionHost
- connectionPort
@@ -86,9 +86,14 @@ spec:
default: false
idempotent:
title: Skip Files Already Read
- description: If true, a file is read only once while the integration
runs, even when it stays on the server. The list of the files read is kept in
memory, so after a restart the files still on the server are read again. If
false, and `delete` is false, the same files are read again on every poll.
+ description: If true, a file is read only once while the integration
runs, even when it stays on the server. The list of the files read is kept in
memory, so after a restart the files still on the server are read again, unless
`move` or `delete` takes them away, or `idempotentRepository` keeps the list
elsewhere. If false, and neither `delete` nor `move` is set, the same files are
read again on every poll.
type: boolean
default: true
+ idempotentRepository:
+ title: Files Read Store
+ description: The name of a bean that keeps the list of the files read,
so that the files left on the server are not read again after a restart, such
as a `FileIdempotentRepository` that keeps the list in a file. Declare the bean
in the integration, for example under `beans` in YAML. Only used when
`idempotent` is true. When it is not set, the list is kept in memory.
+ type: string
+ example: readFiles
binary:
title: Binary Mode
description: If true, files are transferred in binary mode, byte for
byte. When false, they are transferred in ASCII mode, which is meant for text.
Set it to true for files that are not text, such as images, archives or PDF
files.
@@ -101,9 +106,19 @@ spec:
default: true
delete:
title: Delete
- description: If true, each file is deleted from the server once it has
been read and handed on successfully. Use it to read each file only once, also
across restarts.
+ description: If true, each file is deleted from the server once it has
been read and handed on successfully. Use it to read each file only once, also
across restarts. To keep the files, set `move` instead.
type: boolean
default: false
+ move:
+ title: Move After Read
+ description: Where to move each file on the server once it has been
read and handed on successfully, so that it is not read again, also after a
restart. A directory name, such as `.done`, is relative to the directory of the
file. A path that ends with the name of the file, such as
`done/$\{date:now:yyyyMMdd}/$\{file:name}`, is relative to `directoryName`,
where `$\{file:name}` is the name of the file and `$\{date:now:yyyyMMdd}`
today's date. A directory whose name starts with a d [...]
+ type: string
+ example: ".done"
+ moveFailed:
+ title: Move Failed Files To
+ description: Where to move a file on the server when it could not be
handed on, for example because the next step failed, so that it is not read
over and over. The same format as `move`, such as `.error`. When it is not set,
a file that fails stays where it is and is read again at every poll. It can be
set together with `delete` or `move`.
+ type: string
+ example: ".error"
dataTypes:
out:
headers:
@@ -137,6 +152,9 @@ spec:
binary: "{{binary}}"
delete: "{{delete}}"
idempotent: "{{idempotent}}"
+ idempotentRepository: "#bean:{{?idempotentRepository}}"
+ move: "{{?move}}"
+ moveFailed: "{{?moveFailed}}"
passiveMode: "{{passiveMode}}"
password: "RAW({{password}})"
recursive: "{{recursive}}"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/ftps-source.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/ftps-source.kamelet.yaml
index 7e733e423..9564bbcf9 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/ftps-source.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/ftps-source.kamelet.yaml
@@ -37,7 +37,7 @@ spec:
Output: the body is the content of the file, as a stream. The `file` and
`ce-file` headers hold the name of the file, with its path below
`directoryName` when `recursive` is true, such as `2026/orders.csv`.
`CamelFileName` holds the same name, and `CamelFileLength` the size of the file
in bytes.
- Good to know: files are not deleted or moved after they are read, unless
`delete` is true. A file is read only once while the integration runs
(`idempotent`), but the files left on the server are all read again after a
restart; set `delete` to true to read each file only once. It uses explicit
TLS: it connects to the normal FTP port, usually 21, and then switches to TLS.
A server that only accepts implicit TLS, usually on port 990, is not supported.
Files are transferred in ASCII ( [...]
+ Good to know: files are not deleted or moved after they are read, unless
`delete` is true or `move` is set. A file is read only once while the
integration runs (`idempotent`), but the files left on the server are all read
again after a restart; to read each file only once, also across restarts, set
`move` to a directory such as `.done`, set `delete` to true, or keep the list
of the files read in a file with `idempotentRepository`. It uses explicit TLS:
it connects to the normal FTP [...]
required:
- connectionHost
- connectionPort
@@ -86,9 +86,14 @@ spec:
default: false
idempotent:
title: Skip Files Already Read
- description: If true, a file is read only once while the integration
runs, even when it stays on the server. The list of the files read is kept in
memory, so after a restart the files still on the server are read again. If
false, and `delete` is false, the same files are read again on every poll.
+ description: If true, a file is read only once while the integration
runs, even when it stays on the server. The list of the files read is kept in
memory, so after a restart the files still on the server are read again, unless
`move` or `delete` takes them away, or `idempotentRepository` keeps the list
elsewhere. If false, and neither `delete` nor `move` is set, the same files are
read again on every poll.
type: boolean
default: true
+ idempotentRepository:
+ title: Files Read Store
+ description: The name of a bean that keeps the list of the files read,
so that the files left on the server are not read again after a restart, such
as a `FileIdempotentRepository` that keeps the list in a file. Declare the bean
in the integration, for example under `beans` in YAML. Only used when
`idempotent` is true. When it is not set, the list is kept in memory.
+ type: string
+ example: readFiles
binary:
title: Binary Mode
description: If true, files are transferred in binary mode, byte for
byte. When false, they are transferred in ASCII mode, which is meant for text.
Set it to true for files that are not text, such as images, archives or PDF
files.
@@ -101,9 +106,19 @@ spec:
default: true
delete:
title: Delete
- description: If true, each file is deleted from the server once it has
been read and handed on successfully. Use it to read each file only once, also
across restarts.
+ description: If true, each file is deleted from the server once it has
been read and handed on successfully. Use it to read each file only once, also
across restarts. To keep the files, set `move` instead.
type: boolean
default: false
+ move:
+ title: Move After Read
+ description: Where to move each file on the server once it has been
read and handed on successfully, so that it is not read again, also after a
restart. A directory name, such as `.done`, is relative to the directory of the
file. A path that ends with the name of the file, such as
`done/$\{date:now:yyyyMMdd}/$\{file:name}`, is relative to `directoryName`,
where `$\{file:name}` is the name of the file and `$\{date:now:yyyyMMdd}`
today's date. A directory whose name starts with a d [...]
+ type: string
+ example: ".done"
+ moveFailed:
+ title: Move Failed Files To
+ description: Where to move a file on the server when it could not be
handed on, for example because the next step failed, so that it is not read
over and over. The same format as `move`, such as `.error`. When it is not set,
a file that fails stays where it is and is read again at every poll. It can be
set together with `delete` or `move`.
+ type: string
+ example: ".error"
dataTypes:
out:
headers:
@@ -137,6 +152,9 @@ spec:
binary: "{{binary}}"
delete: "{{delete}}"
idempotent: "{{idempotent}}"
+ idempotentRepository: "#bean:{{?idempotentRepository}}"
+ move: "{{?move}}"
+ moveFailed: "{{?moveFailed}}"
passiveMode: "{{passiveMode}}"
password: "RAW({{password}})"
recursive: "{{recursive}}"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/sftp-source.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/sftp-source.kamelet.yaml
index a21c2f781..f4697f038 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/sftp-source.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/sftp-source.kamelet.yaml
@@ -37,7 +37,7 @@ spec:
Output: the body is the content of the file, as bytes. The `file` and
`ce-file` headers hold the name of the file, with its path below
`directoryName` when `recursive` is true, such as `2026/orders.csv`.
`CamelFileName` holds the same name, and `CamelFileLength` the size of the file
in bytes.
- Good to know: files are not deleted or moved after they are read, unless
`delete` is true. A file is read only once while the integration runs
(`idempotent`), but the files left on the server are all read again after a
restart; set `delete` to true to read each file only once. Log in with
`username` and either `password` or a private key. The key of the server must
be in a known_hosts file, `knownHostsUri` or else `~/.ssh/known_hosts` of the
user that runs the integration, or the c [...]
+ Good to know: files are not deleted or moved after they are read, unless
`delete` is true or `move` is set. A file is read only once while the
integration runs (`idempotent`), but the files left on the server are all read
again after a restart; to read each file only once, also across restarts, set
`move` to a directory such as `.done`, set `delete` to true, or keep the list
of the files read in a file with `idempotentRepository`. Log in with `username`
and either `password` or a p [...]
required:
- connectionHost
- connectionPort
@@ -84,9 +84,14 @@ spec:
default: false
idempotent:
title: Skip Files Already Read
- description: If true, a file is read only once while the integration
runs, even when it stays on the server. The list of the files read is kept in
memory, so after a restart the files still on the server are read again. If
false, and `delete` is false, the same files are read again on every poll.
+ description: If true, a file is read only once while the integration
runs, even when it stays on the server. The list of the files read is kept in
memory, so after a restart the files still on the server are read again, unless
`move` or `delete` takes them away, or `idempotentRepository` keeps the list
elsewhere. If false, and neither `delete` nor `move` is set, the same files are
read again on every poll.
type: boolean
default: true
+ idempotentRepository:
+ title: Files Read Store
+ description: The name of a bean that keeps the list of the files read,
so that the files left on the server are not read again after a restart, such
as a `FileIdempotentRepository` that keeps the list in a file. Declare the bean
in the integration, for example under `beans` in YAML. Only used when
`idempotent` is true. When it is not set, the list is kept in memory.
+ type: string
+ example: readFiles
ignoreFileNotFoundOrPermissionError:
title: Ignore Access Errors
description: If true, a directory or a file that does not exist, or
that `username` may not read, is skipped. If false, it causes an error.
@@ -139,9 +144,19 @@ spec:
default: true
delete:
title: Delete
- description: If true, each file is deleted from the server once it has
been read and handed on successfully. Use it to read each file only once, also
across restarts.
+ description: If true, each file is deleted from the server once it has
been read and handed on successfully. Use it to read each file only once, also
across restarts. To keep the files, set `move` instead.
type: boolean
default: false
+ move:
+ title: Move After Read
+ description: Where to move each file on the server once it has been
read and handed on successfully, so that it is not read again, also after a
restart. A directory name, such as `.done`, is relative to the directory of the
file. A path that ends with the name of the file, such as
`done/$\{date:now:yyyyMMdd}/$\{file:name}`, is relative to `directoryName`,
where `$\{file:name}` is the name of the file and `$\{date:now:yyyyMMdd}`
today's date. A directory whose name starts with a d [...]
+ type: string
+ example: ".done"
+ moveFailed:
+ title: Move Failed Files To
+ description: Where to move a file on the server when it could not be
handed on, for example because the next step failed, so that it is not read
over and over. The same format as `move`, such as `.error`. When it is not set,
a file that fails stays where it is and is read again at every poll. It can be
set together with `delete` or `move`.
+ type: string
+ example: ".error"
dataTypes:
out:
headers:
@@ -175,8 +190,11 @@ spec:
binary: "{{binary}}"
delete: "{{delete}}"
idempotent: "{{idempotent}}"
+ idempotentRepository: "#bean:{{?idempotentRepository}}"
ignoreFileNotFoundOrPermissionError:
"{{ignoreFileNotFoundOrPermissionError}}"
knownHostsUri: "RAW({{?knownHostsUri}})"
+ move: "{{?move}}"
+ moveFailed: "{{?moveFailed}}"
passiveMode: "{{passiveMode}}"
password: "RAW({{?password}})"
privateKeyFile: "RAW({{?privateKeyFile}})"