davsclaus opened a new pull request, #3307:
URL: https://github.com/apache/camel-kamelets/pull/3307

   Part of #3117, batch 9b (object store and file Kamelets). Adds the `binary` 
(`application-octet-stream`) data type as the default input of the object store 
sinks and `scp-sink`, and as the default output of 
`langchain4j-ingest-file-source`. Camel runs the transformer of a default on 
every message, so each change was checked with `camel run` (the body class and 
`Content-Type` before and after, against `main`, with stream caching on and 
off) and with the Citrus tests.
   
   **Draft:** this needs a Camel snapshot with CAMEL-27266 and CAMEL-27267 
(merged in Camel main today). With the snapshot CI uses now, `binary` reads a 
`File` or a `GenericFile` body into memory. See _Needs_.
   
   ## What changes
   
   | Kamelet | in | out | What changes (before → after) |
   |---|---|---|---|
   | `aws-s3-sink`, `ceph-sink`, `minio-sink`, `ibm-cos-sink`, 
`azure-storage-files-sink`, `scp-sink` | `binary` | - | Text → `byte[]` before 
the upload. `Content-Type` after the sink: as it came → 
`application/octet-stream`. A `byte[]`, a `File`, a `GenericFile` and a cached 
stream are kept as they are. A `Map` fails at the start instead of in the 
component (it failed before too). The stored data does not change. |
   | `azure-storage-blob-sink`, `azure-storage-blob-append-sink`, 
`azure-storage-datalake-sink` | `binary` | - | As above. The body afterwards is 
still `true`; `Content-Type` is now `application/octet-stream`. |
   | `google-storage-sink` | `binary` | - | As above. The body afterwards still 
describes the uploaded file; `Content-Type` is now `application/octet-stream`. |
   | `langchain4j-ingest-file-source` | - | `binary` | `Content-Type`: none → 
`application/octet-stream`. The body stays the file (`GenericFile`), also with 
`charset`: it is not read into memory. |
   
   Measured with `camel run` and the components stubbed, for all ten sinks 
(same result for each):
   
   | body sent | before | after, stream caching on | after, stream caching off |
   |---|---|---|---|
   | `String` | `String` | `byte[]` | `byte[]` |
   | `byte[]` | `byte[]` | `byte[]` | `byte[]` |
   | `java.io.File` | `File` | `File` | `File` |
   | `GenericFile` (from `file:`) | `GenericFile` | `GenericFile` | 
`GenericFile` |
   | `InputStream` | `ByteArrayInputStreamCache` | `ByteArrayInputStreamCache` 
| `byte[]` (was `ByteArrayInputStream`) |
   | `Map` | `LinkedHashMap` (the component then fails) | fails: no converter 
to `byte[]` | fails |
   | null | null, no `Content-Type` | null, no `Content-Type` | null, no 
`Content-Type` |
   
   With stream caching off a plain stream is read into a `byte[]`. For these 
ten sinks that adds no memory: each component reads a stream of unknown length 
into memory, or into stream caching, anyway (`PayloadHelper.cacheStream` in 
aws2-s3, minio, ibm-cos, google-storage, azure blob and Data Lake; Azure Files 
converts it to `byte[]` when `CamelFileLength` is missing, which the template 
removes; SCP buffers the whole stream to send its size first).
   
   **Content-Type sent to the remote side (decision D2).** None of the 
components reads the `Content-Type` header, so the content type the file is 
stored with does not change:
   - `aws2-s3` (`aws-s3-sink`, `ceph-sink`): only `CamelAwsS3ContentType`, 
which `ceph-sink` removes.
   - `minio`: only `CamelMinioContentType`, which the template removes.
   - `ibm-cos`: only `CamelIBMCOSContentType`, which the template removes.
   - `google-storage`: only `CamelGoogleCloudStorageContentType`, which the 
template removes.
   - `azure-storage-blob`: only `CamelAzureStorageBlobHttpHeaders`, which the 
template removes.
   - `azure-storage-datalake`: only its `PathHttpHeaders` header, which the 
template removes.
   - `azure-files`, `scp`: no content type.
   
   The `ceph-sink` test now sends `Content-Type: text/csv` and checks that the 
stored object has the S3 client default `application/octet-stream`.
   
   The `Input:` and `Output:` paragraphs of the eleven Kamelets were updated.
   
   ## Not done
   
   - **`ftp-sink`, `ftps-sink`, `sftp-sink` (in):** the FTP and SFTP clients 
send a stream to the server as they read it. With stream caching off, `binary` 
would read the stream into memory first, such as a large file from `ftp-source` 
or `sftp-source` with `streamDownload`, which their description recommends with 
stream caching off. Above 2 GB that fails. A YAML comment gives the reason.
   - **`ftp-source`, `ftps-source`, `sftp-source` (out), again:** CAMEL-27267 
keeps a `GenericFile`, but these templates do not hand on the file. 
`ftp-source` converts the body to an `InputStream`, and `sftp-source` takes the 
content out of the file. So `binary` still reads a `streamDownload` file into 
memory when stream caching is off. Measured with `camel run` against the 
embedded FTP and SFTP servers of `camel infra`:
   
   | Kamelet, mode | stream caching | before | with `binary` |
   |---|---|---|---|
   | `ftp-source`, default | on | `ByteArrayInputStreamCache` | 
`ByteArrayInputStreamCache` |
   | `ftp-source`, default | off | `ByteArrayInputStream` | `byte[]` |
   | `ftp-source`, `streamDownload` | on | `InputStreamCache` | 
`InputStreamCache` |
   | `ftp-source`, `streamDownload` | off | `SocketInputStream` | **`byte[]`: 
whole file in memory** |
   | `sftp-source`, default | on / off | `byte[]` | `byte[]` |
   | `sftp-source`, `streamDownload` | on | `InputStreamCache` | 
`InputStreamCache` |
   | `sftp-source`, `streamDownload` | off | `ChannelSftp$2` | **`byte[]`: 
whole file in memory** |
   
     Two ways out: `application-octet-stream` keeps any `InputStream` (a Camel 
change), or the templates hand on the file and leave reading it to the next 
step. Until then, a YAML comment gives the reason.
   - **`azure-storage-files-source`:** it has had `out: binary` since #1965, so 
nothing to add. With CAMEL-27267 its body is now the file (a `GenericFile`), 
not a `byte[]`. #3304 changes its `Output:` to "as bytes", which is only true 
with the older snapshot. Not checked by running it: Azurite has no Azure Files. 
The same transformer keeps the `GenericFile` of 
`langchain4j-ingest-file-source`.
   
   ## Tests
   
   Run locally against Camel 4.24.0-SNAPSHOT built from Camel main today (with 
CAMEL-27258, CAMEL-27266, CAMEL-27267).
   - `./mvnw -o verify -pl :camel-kamelets-itest -Denable.integration.tests 
-Dit.test='FileIT,CloudIT#ceph+minio,AwsIT#awsS3,AzureIT,GoogleCloudIT,ServicesIT#ibmCos,CommonIT#langchain4jIngest'`:
 55 tests run, 46 pass and 9 fail.
     - Every sink test passes: `aws-s3-sink`, `azure-storage-blob-sink`, 
`azure-storage-blob-append-sink`, `azure-storage-datalake-sink`, 
`google-storage-sink`, `ceph-sink`, `minio-sink`, `ibm-cos-sink`, `scp-sink`, 
and the unchanged `ftp-sink`, `ftps-sink`, `sftp-sink`. FileIT: 10 of 10. 
CloudIT: 4 of 4. AwsIT `aws.s3`: 7 of 7.
     - 7 failures are source tests this PR does not touch: 
`azure-storage-blob-source`, `azure-storage-datalake-source`, 
`azure-storage-queue-source`, `azure-eventhubs-source`, 
`azure-cosmosdb-source`, `google-storage-source` and `ibm-cos-source`. Their 
existing `out` defaults now reach the consuming route (CAMEL-27258), and #3304 
updates exactly these tests.
     - 2 failures were in this PR's tests and are fixed: 
`parser-max-size-route` and `langchain4j-ingest-file-media-route` sent the new 
`Content-Type: application/octet-stream` on to the Citrus HTTP server. Run 
again, `CommonIT#langchain4jIngest`: 11 of 11.
     - The Azure Files sink has no test (no emulator). Its row comes from the 
stubbed `camel run`.
   - The sink routes now record the body class and `Content-Type` after the 
Kamelet: `[B` / `ct=application/octet-stream`, or `java.lang.Boolean` and 
`com.google.cloud.storage.Blob` where the component sets the body. They remove 
`Content-Type` before the HTTP call, because Citrus reads an 
`application/octet-stream` body as bytes. `aws-s3-sink-route` logs `Kamelet 
result: body=[B ct=application/octet-stream` and the test verifies that line.
   - `langchain4j-ingest-file-source-route` checks 
`org.apache.camel.component.file.GenericFile` and `ct=application/octet-stream`.
   - `./mvnw clean install` from the root, twice: both BUILD SUCCESS.
   
   ## Overlap
   
   - #3304 also changes the `azure-storage-files-source` description (see _Not 
done_). This PR does not touch that Kamelet.
   - #3292 (FTPS trust store options) and #3285 (`google-storage-sink` option 
rename) change `ftps-source`, `ftps-sink` and `google-storage-sink`. Here those 
Kamelets only get a YAML comment or a `dataTypes` block and an 
`Input:`/`Output:` sentence, so whichever merges second needs a small rebase.
   - #3302 (MinIO and SSH sources) dropped the FTP and SFTP sources for the 
same reason given above. No file overlap.
   - #3294 (description wording) may touch the same `Input:`/`Output:` lines: a 
rebase.
   
   ## Needs
   
   A Camel 4.24.0-SNAPSHOT with CAMEL-27266 and CAMEL-27267, expected tonight. 
Before that, `binary` reads a `File` or `GenericFile` into a `byte[]`, so the 
`langchain4j-ingest-file-source` test fails in CI, and a file sent to the sinks 
is read into memory.
   
   ## Upgrade note for #3190 (Data types section)
   
   - **Default data types on the object store and file Kamelets (#PR):**
     - `aws-s3-sink`, `ceph-sink`, `minio-sink`, `ibm-cos-sink`, 
`google-storage-sink`, `azure-storage-blob-sink`, 
`azure-storage-blob-append-sink`, `azure-storage-datalake-sink`, 
`azure-storage-files-sink` and `scp-sink` (`in: binary`) turn text into bytes 
before the upload. A file and a cached stream are kept as they are. With stream 
caching off, a stream is read into bytes first. These components already did 
that for a stream of unknown length.
     - After these sinks, `Content-Type` is `application/octet-stream`, and a 
text body is now bytes. The content type the file is stored with does not 
change: these sinks never used `Content-Type`.
     - A `Map` body now fails at the start of these sinks. It failed in the 
component before.
     - `langchain4j-ingest-file-source` (`out: binary`) sets `Content-Type: 
application/octet-stream` on each file. The body is still the file. A sink that 
forwards `Content-Type`, such as `http-sink`, now sends it on, also after 
`langchain4j-ingest-sink`. Remove the header first if that is not wanted.
     - `ftp-sink`, `ftps-sink`, `sftp-sink`, `ftp-source`, `ftps-source` and 
`sftp-source` get no default data type, so large files keep streaming with 
stream caching off.
   
   _Claude Code on behalf of Claus Ibsen_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to