davsclaus opened a new pull request, #3307:
URL: https://github.com/apache/camel-kamelets/pull/3307
Part of #3117, batch 9b (object store and file Kamelets). Adds the `binary`
(`application-octet-stream`) data type as the default input of the object store
sinks and `scp-sink`, and as the default output of
`langchain4j-ingest-file-source`. Camel runs the transformer of a default on
every message, so each change was checked with `camel run` (the body class and
`Content-Type` before and after, against `main`, with stream caching on and
off) and with the Citrus tests.
**Draft:** this needs a Camel snapshot with CAMEL-27266 and CAMEL-27267
(merged in Camel main today). With the snapshot CI uses now, `binary` reads a
`File` or a `GenericFile` body into memory. See _Needs_.
## What changes
| Kamelet | in | out | What changes (before → after) |
|---|---|---|---|
| `aws-s3-sink`, `ceph-sink`, `minio-sink`, `ibm-cos-sink`,
`azure-storage-files-sink`, `scp-sink` | `binary` | - | Text → `byte[]` before
the upload. `Content-Type` after the sink: as it came →
`application/octet-stream`. A `byte[]`, a `File`, a `GenericFile` and a cached
stream are kept as they are. A `Map` fails at the start instead of in the
component (it failed before too). The stored data does not change. |
| `azure-storage-blob-sink`, `azure-storage-blob-append-sink`,
`azure-storage-datalake-sink` | `binary` | - | As above. The body afterwards is
still `true`; `Content-Type` is now `application/octet-stream`. |
| `google-storage-sink` | `binary` | - | As above. The body afterwards still
describes the uploaded file; `Content-Type` is now `application/octet-stream`. |
| `langchain4j-ingest-file-source` | - | `binary` | `Content-Type`: none →
`application/octet-stream`. The body stays the file (`GenericFile`), also with
`charset`: it is not read into memory. |
Measured with `camel run` and the components stubbed, for all ten sinks
(same result for each):
| body sent | before | after, stream caching on | after, stream caching off |
|---|---|---|---|
| `String` | `String` | `byte[]` | `byte[]` |
| `byte[]` | `byte[]` | `byte[]` | `byte[]` |
| `java.io.File` | `File` | `File` | `File` |
| `GenericFile` (from `file:`) | `GenericFile` | `GenericFile` |
`GenericFile` |
| `InputStream` | `ByteArrayInputStreamCache` | `ByteArrayInputStreamCache`
| `byte[]` (was `ByteArrayInputStream`) |
| `Map` | `LinkedHashMap` (the component then fails) | fails: no converter
to `byte[]` | fails |
| null | null, no `Content-Type` | null, no `Content-Type` | null, no
`Content-Type` |
With stream caching off a plain stream is read into a `byte[]`. For these
ten sinks that adds no memory: each component reads a stream of unknown length
into memory, or into stream caching, anyway (`PayloadHelper.cacheStream` in
aws2-s3, minio, ibm-cos, google-storage, azure blob and Data Lake; Azure Files
converts it to `byte[]` when `CamelFileLength` is missing, which the template
removes; SCP buffers the whole stream to send its size first).
**Content-Type sent to the remote side (decision D2).** None of the
components reads the `Content-Type` header, so the content type the file is
stored with does not change:
- `aws2-s3` (`aws-s3-sink`, `ceph-sink`): only `CamelAwsS3ContentType`,
which `ceph-sink` removes.
- `minio`: only `CamelMinioContentType`, which the template removes.
- `ibm-cos`: only `CamelIBMCOSContentType`, which the template removes.
- `google-storage`: only `CamelGoogleCloudStorageContentType`, which the
template removes.
- `azure-storage-blob`: only `CamelAzureStorageBlobHttpHeaders`, which the
template removes.
- `azure-storage-datalake`: only its `PathHttpHeaders` header, which the
template removes.
- `azure-files`, `scp`: no content type.
The `ceph-sink` test now sends `Content-Type: text/csv` and checks that the
stored object has the S3 client default `application/octet-stream`.
The `Input:` and `Output:` paragraphs of the eleven Kamelets were updated.
## Not done
- **`ftp-sink`, `ftps-sink`, `sftp-sink` (in):** the FTP and SFTP clients
send a stream to the server as they read it. With stream caching off, `binary`
would read the stream into memory first, such as a large file from `ftp-source`
or `sftp-source` with `streamDownload`, which their description recommends with
stream caching off. Above 2 GB that fails. A YAML comment gives the reason.
- **`ftp-source`, `ftps-source`, `sftp-source` (out), again:** CAMEL-27267
keeps a `GenericFile`, but these templates do not hand on the file.
`ftp-source` converts the body to an `InputStream`, and `sftp-source` takes the
content out of the file. So `binary` still reads a `streamDownload` file into
memory when stream caching is off. Measured with `camel run` against the
embedded FTP and SFTP servers of `camel infra`:
| Kamelet, mode | stream caching | before | with `binary` |
|---|---|---|---|
| `ftp-source`, default | on | `ByteArrayInputStreamCache` |
`ByteArrayInputStreamCache` |
| `ftp-source`, default | off | `ByteArrayInputStream` | `byte[]` |
| `ftp-source`, `streamDownload` | on | `InputStreamCache` |
`InputStreamCache` |
| `ftp-source`, `streamDownload` | off | `SocketInputStream` | **`byte[]`:
whole file in memory** |
| `sftp-source`, default | on / off | `byte[]` | `byte[]` |
| `sftp-source`, `streamDownload` | on | `InputStreamCache` |
`InputStreamCache` |
| `sftp-source`, `streamDownload` | off | `ChannelSftp$2` | **`byte[]`:
whole file in memory** |
Two ways out: `application-octet-stream` keeps any `InputStream` (a Camel
change), or the templates hand on the file and leave reading it to the next
step. Until then, a YAML comment gives the reason.
- **`azure-storage-files-source`:** it has had `out: binary` since #1965, so
nothing to add. With CAMEL-27267 its body is now the file (a `GenericFile`),
not a `byte[]`. #3304 changes its `Output:` to "as bytes", which is only true
with the older snapshot. Not checked by running it: Azurite has no Azure Files.
The same transformer keeps the `GenericFile` of
`langchain4j-ingest-file-source`.
## Tests
Run locally against Camel 4.24.0-SNAPSHOT built from Camel main today (with
CAMEL-27258, CAMEL-27266, CAMEL-27267).
- `./mvnw -o verify -pl :camel-kamelets-itest -Denable.integration.tests
-Dit.test='FileIT,CloudIT#ceph+minio,AwsIT#awsS3,AzureIT,GoogleCloudIT,ServicesIT#ibmCos,CommonIT#langchain4jIngest'`:
55 tests run, 46 pass and 9 fail.
- Every sink test passes: `aws-s3-sink`, `azure-storage-blob-sink`,
`azure-storage-blob-append-sink`, `azure-storage-datalake-sink`,
`google-storage-sink`, `ceph-sink`, `minio-sink`, `ibm-cos-sink`, `scp-sink`,
and the unchanged `ftp-sink`, `ftps-sink`, `sftp-sink`. FileIT: 10 of 10.
CloudIT: 4 of 4. AwsIT `aws.s3`: 7 of 7.
- 7 failures are source tests this PR does not touch:
`azure-storage-blob-source`, `azure-storage-datalake-source`,
`azure-storage-queue-source`, `azure-eventhubs-source`,
`azure-cosmosdb-source`, `google-storage-source` and `ibm-cos-source`. Their
existing `out` defaults now reach the consuming route (CAMEL-27258), and #3304
updates exactly these tests.
- 2 failures were in this PR's tests and are fixed:
`parser-max-size-route` and `langchain4j-ingest-file-media-route` sent the new
`Content-Type: application/octet-stream` on to the Citrus HTTP server. Run
again, `CommonIT#langchain4jIngest`: 11 of 11.
- The Azure Files sink has no test (no emulator). Its row comes from the
stubbed `camel run`.
- The sink routes now record the body class and `Content-Type` after the
Kamelet: `[B` / `ct=application/octet-stream`, or `java.lang.Boolean` and
`com.google.cloud.storage.Blob` where the component sets the body. They remove
`Content-Type` before the HTTP call, because Citrus reads an
`application/octet-stream` body as bytes. `aws-s3-sink-route` logs `Kamelet
result: body=[B ct=application/octet-stream` and the test verifies that line.
- `langchain4j-ingest-file-source-route` checks
`org.apache.camel.component.file.GenericFile` and `ct=application/octet-stream`.
- `./mvnw clean install` from the root, twice: both BUILD SUCCESS.
## Overlap
- #3304 also changes the `azure-storage-files-source` description (see _Not
done_). This PR does not touch that Kamelet.
- #3292 (FTPS trust store options) and #3285 (`google-storage-sink` option
rename) change `ftps-source`, `ftps-sink` and `google-storage-sink`. Here those
Kamelets only get a YAML comment or a `dataTypes` block and an
`Input:`/`Output:` sentence, so whichever merges second needs a small rebase.
- #3302 (MinIO and SSH sources) dropped the FTP and SFTP sources for the
same reason given above. No file overlap.
- #3294 (description wording) may touch the same `Input:`/`Output:` lines: a
rebase.
## Needs
A Camel 4.24.0-SNAPSHOT with CAMEL-27266 and CAMEL-27267, expected tonight.
Before that, `binary` reads a `File` or `GenericFile` into a `byte[]`, so the
`langchain4j-ingest-file-source` test fails in CI, and a file sent to the sinks
is read into memory.
## Upgrade note for #3190 (Data types section)
- **Default data types on the object store and file Kamelets (#PR):**
- `aws-s3-sink`, `ceph-sink`, `minio-sink`, `ibm-cos-sink`,
`google-storage-sink`, `azure-storage-blob-sink`,
`azure-storage-blob-append-sink`, `azure-storage-datalake-sink`,
`azure-storage-files-sink` and `scp-sink` (`in: binary`) turn text into bytes
before the upload. A file and a cached stream are kept as they are. With stream
caching off, a stream is read into bytes first. These components already did
that for a stream of unknown length.
- After these sinks, `Content-Type` is `application/octet-stream`, and a
text body is now bytes. The content type the file is stored with does not
change: these sinks never used `Content-Type`.
- A `Map` body now fails at the start of these sinks. It failed in the
component before.
- `langchain4j-ingest-file-source` (`out: binary`) sets `Content-Type:
application/octet-stream` on each file. The body is still the file. A sink that
forwards `Content-Type`, such as `http-sink`, now sends it on, also after
`langchain4j-ingest-sink`. Remove the header first if that is not wanted.
- `ftp-sink`, `ftps-sink`, `sftp-sink`, `ftp-source`, `ftps-source` and
`sftp-source` get no default data type, so large files keep streaming with
stream caching off.
_Claude Code on behalf of Claus Ibsen_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]