davsclaus opened a new pull request, #3302: URL: https://github.com/apache/camel-kamelets/pull/3302
Part of #3117, batch 3 (storage, files, shell) of the data type proposal. **Draft:** a source `out` default runs its transformer when the message is handed to `kamelet:sink` only with CAMEL-27258 (merged in Camel main today). The Camel snapshot CI uses predates it, so the changed Citrus expectations fail in CI until a snapshot with CAMEL-27258 is published. Locally (Camel main with CAMEL-27258) they pass. ## What changes | Kamelet | out | what changes (before → after) | memory effect | |---|---|---|---| | `minio-source` | `binary` (`application-octet-stream`) | body `byte[]` → `byte[]` (unchanged); `Content-Type` none → `application/octet-stream` | none: the consumer has already read the object into a `byte[]`; the transformer only sets the header | | `ssh-source` | `text` (`text-plain`) | body `String` → `String` (unchanged); `Content-Type` none → `text/plain` | none: the template already converts the output to a `String` | The `Output:` paragraphs of both now say which `Content-Type` is set. The minio one also names `CamelMinioContentType` as the header that keeps the content type the file was stored with. ## Dropped from the batch: `ftp-source`, `ftps-source`, `sftp-source`, `ceph-source` Each of them has a mode meant for files too large for memory (`streamDownload=true` on the FTP/SFTP sources, `includeBody=false` on `ceph-source`). With stream caching off, `binary` would read the whole file into a `byte[]` where today it is handed on as the live download stream. Measured with Citrus (body class in the consuming route, Camel main with CAMEL-27258): | Kamelet, mode | stream caching | before | with `binary` | |---|---|---|---| | `ftp-source`, default | on | `ByteArrayInputStreamCache` | `ByteArrayInputStreamCache` + `application/octet-stream` | | `ftp-source`, default | off | `ByteArrayInputStream` | `byte[]` (already in memory) | | `ftp-source`, `streamDownload=true` | on | `InputStreamCache` | `InputStreamCache` | | `ftp-source`, `streamDownload=true` | off | `org.apache.commons.net.io.SocketInputStream` | **`byte[]`: whole file in memory** | | `sftp-source`, default | on | `byte[]` | `byte[]` | | `sftp-source`, `streamDownload=true` | on | `InputStreamCache` | `InputStreamCache` | | `sftp-source`, `streamDownload=true` | off | `com.jcraft.jsch.ChannelSftp$2` | **`byte[]`: whole file in memory** | | `ceph-source`, default | on | `byte[]` | `byte[]` | | `ceph-source`, `includeBody=false` | on | `InputStreamCache` | `InputStreamCache` | | `ceph-source`, `includeBody=false` | off | `software.amazon.awssdk.core.ResponseInputStream` | **`byte[]`: whole object in memory** | The `streamDownload` description of the FTP sources recommends turning stream caching off for large files, so the default would defeat a documented setup. These four wait for a Camel change: `application-octet-stream` keeps a plain `InputStream` (or keeps it when stream caching is off), next to C3 (keep a `File`/`WrappedFile`). `ftps-source` is the same template as `ftp-source` (its test behaved the same). ## Tests - `minio-source-route` and `ssh-source-route` expect the new `Content-Type`. The minio route removes `Content-Type` after recording it, because the Citrus HTTP server reads an `application/octet-stream` body as bytes and the test checks the content as text. - New `ssh/sftp-source-stream-route`: `sftp-source` with `streamDownload=true` gives a stream (`InputStreamCache` with stream caching on), as the batch row asked. It records today's behaviour and will catch a future default that reads the stream into memory. - `./mvnw -o verify -pl tests/camel-kamelets-itest -Dit.test='CloudIT#ceph+minio,FileIT'`: 15 tests, all pass (FileIT 11, CloudIT 4), locally with Camel main including CAMEL-27258. The FTPS source test passed too. - `./mvnw clean install` from the root, twice: both BUILD SUCCESS. ## Overlap - #3292 (FTPS trust store options) and #3284 (FTPS test trust store) change `ftps-source` and the FTPS tests. This PR no longer touches either, so there is no conflict. - #3294 changes descriptions, but not `minio-source` or `ssh-source`. ## Upgrade note for #3190 (ready to copy) ``` === Kamelets: minio-source and ssh-source set a Content-Type header `minio-source` now declares the output data type `binary` (`application-octet-stream`) and `ssh-source` the output data type `text` (`text-plain`). Each message they hand on now has the `Content-Type` header set to `application/octet-stream` (`minio-source`) or `text/plain` (`ssh-source`). The body is unchanged: the content of the file as bytes, and the output of the command as text. A sink that forwards `Content-Type`, such as `http-sink` or `kafka-sink`, now sends it on; remove the header before the sink if that is not wanted. The content type a MinIO file was stored with stays in the `CamelMinioContentType` header. ``` _Claude Code on behalf of Claus Ibsen_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_016Y4dDmBDsSWV117kccuNf9 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
