davsclaus opened a new pull request, #3302:
URL: https://github.com/apache/camel-kamelets/pull/3302

   Part of #3117, batch 3 (storage, files, shell) of the data type proposal.
   
   **Draft:** a source `out` default runs its transformer when the message is 
handed to `kamelet:sink` only with CAMEL-27258 (merged in Camel main today). 
The Camel snapshot CI uses predates it, so the changed Citrus expectations fail 
in CI until a snapshot with CAMEL-27258 is published. Locally (Camel main with 
CAMEL-27258) they pass.
   
   ## What changes
   
   | Kamelet | out | what changes (before → after) | memory effect |
   |---|---|---|---|
   | `minio-source` | `binary` (`application-octet-stream`) | body `byte[]` → 
`byte[]` (unchanged); `Content-Type` none → `application/octet-stream` | none: 
the consumer has already read the object into a `byte[]`; the transformer only 
sets the header |
   | `ssh-source` | `text` (`text-plain`) | body `String` → `String` 
(unchanged); `Content-Type` none → `text/plain` | none: the template already 
converts the output to a `String` |
   
   The `Output:` paragraphs of both now say which `Content-Type` is set. The 
minio one also names `CamelMinioContentType` as the header that keeps the 
content type the file was stored with.
   
   ## Dropped from the batch: `ftp-source`, `ftps-source`, `sftp-source`, 
`ceph-source`
   
   Each of them has a mode meant for files too large for memory 
(`streamDownload=true` on the FTP/SFTP sources, `includeBody=false` on 
`ceph-source`). With stream caching off, `binary` would read the whole file 
into a `byte[]` where today it is handed on as the live download stream. 
Measured with Citrus (body class in the consuming route, Camel main with 
CAMEL-27258):
   
   | Kamelet, mode | stream caching | before | with `binary` |
   |---|---|---|---|
   | `ftp-source`, default | on | `ByteArrayInputStreamCache` | 
`ByteArrayInputStreamCache` + `application/octet-stream` |
   | `ftp-source`, default | off | `ByteArrayInputStream` | `byte[]` (already 
in memory) |
   | `ftp-source`, `streamDownload=true` | on | `InputStreamCache` | 
`InputStreamCache` |
   | `ftp-source`, `streamDownload=true` | off | 
`org.apache.commons.net.io.SocketInputStream` | **`byte[]`: whole file in 
memory** |
   | `sftp-source`, default | on | `byte[]` | `byte[]` |
   | `sftp-source`, `streamDownload=true` | on | `InputStreamCache` | 
`InputStreamCache` |
   | `sftp-source`, `streamDownload=true` | off | 
`com.jcraft.jsch.ChannelSftp$2` | **`byte[]`: whole file in memory** |
   | `ceph-source`, default | on | `byte[]` | `byte[]` |
   | `ceph-source`, `includeBody=false` | on | `InputStreamCache` | 
`InputStreamCache` |
   | `ceph-source`, `includeBody=false` | off | 
`software.amazon.awssdk.core.ResponseInputStream` | **`byte[]`: whole object in 
memory** |
   
   The `streamDownload` description of the FTP sources recommends turning 
stream caching off for large files, so the default would defeat a documented 
setup. These four wait for a Camel change: `application-octet-stream` keeps a 
plain `InputStream` (or keeps it when stream caching is off), next to C3 (keep 
a `File`/`WrappedFile`). `ftps-source` is the same template as `ftp-source` 
(its test behaved the same).
   
   ## Tests
   
   - `minio-source-route` and `ssh-source-route` expect the new `Content-Type`. 
The minio route removes `Content-Type` after recording it, because the Citrus 
HTTP server reads an `application/octet-stream` body as bytes and the test 
checks the content as text.
   - New `ssh/sftp-source-stream-route`: `sftp-source` with 
`streamDownload=true` gives a stream (`InputStreamCache` with stream caching 
on), as the batch row asked. It records today's behaviour and will catch a 
future default that reads the stream into memory.
   - `./mvnw -o verify -pl tests/camel-kamelets-itest 
-Dit.test='CloudIT#ceph+minio,FileIT'`: 15 tests, all pass (FileIT 11, CloudIT 
4), locally with Camel main including CAMEL-27258. The FTPS source test passed 
too.
   - `./mvnw clean install` from the root, twice: both BUILD SUCCESS.
   
   ## Overlap
   
   - #3292 (FTPS trust store options) and #3284 (FTPS test trust store) change 
`ftps-source` and the FTPS tests. This PR no longer touches either, so there is 
no conflict.
   - #3294 changes descriptions, but not `minio-source` or `ssh-source`.
   
   ## Upgrade note for #3190 (ready to copy)
   
   ```
   === Kamelets: minio-source and ssh-source set a Content-Type header
   
   `minio-source` now declares the output data type `binary` 
(`application-octet-stream`) and `ssh-source` the
   output data type `text` (`text-plain`). Each message they hand on now has 
the `Content-Type` header set to
   `application/octet-stream` (`minio-source`) or `text/plain` (`ssh-source`). 
The body is unchanged: the content
   of the file as bytes, and the output of the command as text. A sink that 
forwards `Content-Type`, such as
   `http-sink` or `kafka-sink`, now sends it on; remove the header before the 
sink if that is not wanted. The
   content type a MinIO file was stored with stays in the 
`CamelMinioContentType` header.
   ```
   
   _Claude Code on behalf of Claus Ibsen_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   
   https://claude.ai/code/session_016Y4dDmBDsSWV117kccuNf9
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to