davsclaus opened a new pull request, #3227:
URL: https://github.com/apache/camel-kamelets/pull/3227

   Part of #3170: the Google Pub/Sub Kamelets can leave the acknowledgement to 
the route, stop processing a message that keeps failing, and publish in order; 
`google-storage-source` can move the files it reads to another bucket, skip 
their content or the folder markers, and poll less often; the Cloud Storage 
Kamelets can log in with Workload Identity Federation; `google-storage-sink` 
can write to a fixed object name and choose where a bucket it creates is 
located; and `google-bigquery-sink` can send an insert ID, so that BigQuery can 
drop the rows a retried message sends again.
   
   ### The options
   
   | Kamelet | Option | Template mapping | What it does |
   |---|---|---|---|
   | `google-pubsub-sink` | `messageOrderingEnabled` | 
`messageOrderingEnabled={{?messageOrderingEnabled}}` | Publishes the messages 
with the same `CamelGooglePubsubOrderingKey` header in order. Without it, a 
message with that header fails. |
   | | `pubsubEndpoint` | `pubsubEndpoint={{?pubsubEndpoint}}` | The Pub/Sub 
service to publish to, such as `europe-west1-pubsub.googleapis.com:443`. 
Ordering needs a regional endpoint. |
   | `google-pubsub-source` | `ackMode` | `ackMode={{?ackMode}}` | `AUTO` 
(unset) or `NONE`: the route acknowledges each message itself through the 
`CamelGooglePubsubAcknowledge` header. |
   | | `maxDeliveryAttempts` | `maxDeliveryAttempts={{?maxDeliveryAttempts}}` | 
A message delivered this many times is given back (nack) without being 
processed. |
   | | `maxAckExtensionPeriod` | 
`maxAckExtensionPeriod={{?maxAckExtensionPeriod}}` | How long, in seconds, an 
unacknowledged message is kept before Pub/Sub delivers it again (streaming 
pull). |
   | `google-storage-event-based-source` | `ackMode`, `maxDeliveryAttempts` | 
as for `google-pubsub-source` | The same, for the bucket notifications. |
   | `google-storage-source` | `moveAfterRead`, `destinationBucket` | 
`moveAfterRead={{?moveAfterRead}}`, `destinationBucket={{?destinationBucket}}` 
| Copies each file to `destinationBucket` once processed, then deletes it 
(unless `deleteAfterRead` is false). |
   | | `includeBody` | `includeBody={{?includeBody}}`, plus a precondition step 
(below) | `false`: the content is not downloaded; the body is empty, the 
headers describe the file. |
   | | `includeFolders` | `includeFolders={{?includeFolders}}` | `false`: skips 
folder markers (objects whose name ends with `/`). |
   | | `delay` | `delay={{?delay}}` | Milliseconds between two polls; 500 when 
unset. |
   | `google-storage-source`, `google-storage-sink` | 
`useWorkloadIdentityFederation`, `workloadIdentityConfig`, 
`impersonatedServiceAccount` | `{{?...}}` each | Log in with Workload Identity 
Federation: the `external_account` configuration file, optionally impersonating 
a service account. |
   | `google-storage-sink` | `objectName` | `objectName=RAW({{?objectName}})` | 
Every message is written to this object; the `file` and `ce-file` headers are 
then not used. |
   | | `storageLocation` | `storageLocation={{?storageLocation}}` | Location of 
a bucket that `autoCreateBucket` creates; `US-EAST1` when unset. |
   | `google-bigquery-sink` | `useAsInsertId` | `useAsInsertId: 
"{{?useAsInsertId}}"` | The field of each row whose value is sent as the insert 
ID of the row. |
   
   - **Defaults:** every option is optional, with no `default`, as in #3174. 
When they are unset, the optional placeholders drop them, and the endpoint URIs 
are identical to `main`'s.
   - **Headers:** the new headers `CamelGooglePubsubOrderingKey` (sink, input), 
`CamelGooglePubsubDeliveryAttempt` and `CamelGooglePubsubAcknowledge` (sources, 
output) are documented in `dataTypes`. `KameletsCatalogTest` counts: 
`google-pubsub-sink` 1 → 2, `google-pubsub-source` 3 → 5.
   - **`includeBody: false` needs a template step.** The component then puts 
the `com.google.cloud.storage.Blob` object of the Google client in the body. 
The `binary` output data type of `google-storage-source` cannot convert it to 
bytes, so every file failed and, not being deleted, was read again on every 
poll. The template now has a `choice` with `precondition: true` on 
`${properties:includeBody:true} == 'false'`, which empties the body. Being a 
precondition, it is resolved once when the route is created, and drops out 
entirely when `includeBody` is unset or true.
   - **`objectName` in `RAW()`:** a name such as `status/latest 100%+x.json` is 
stored as written.
   - **Patterns:** `pubsubEndpoint` (`host:port`), `destinationBucket` (bucket 
name), `storageLocation`, `objectName` (no line breaks), 
`workloadIdentityConfig` (`file:`, `classpath:`, `http:`, `https:`), 
`impersonatedServiceAccount` (an email), `useAsInsertId` (a column name).
   - **Descriptions:** "Good to know" of each Kamelet names the new options. 
The partial pages have new sections with examples: Message Ordering; 
Acknowledging Messages and Dead-Letter Topic; Acknowledging Notifications; 
Moving Files to Another Bucket and Workload Identity Federation; A Fixed Object 
Name and Creating the Bucket; Insert IDs.
   
   ### Left out
   
   - **`storageClass` on `google-storage-sink`.** The component option is typed 
`com.google.cloud.storage.StorageClass`, and Camel has no type converter from 
text to it: `storageClass=NEARLINE` fails the route at start with `No type 
converter available to convert from type: java.lang.String to the required 
type: com.google.cloud.storage.StorageClass`. It needs a Camel change first, 
such as a type converter in `camel-google-storage`. Also, the component only 
uses it for a bucket that `autoCreateBucket` creates: the upload request sets 
no storage class, so each file gets the default class of its bucket. A storage 
class per file would need a producer change too.
   - **No emulator option on the Kamelets.** None is needed for testing: 
Pub/Sub is pointed at an emulator with the component options 
`camel.component.google-pubsub.endpoint` and 
`camel.component.google-pubsub.authenticate=false`. The Java Cloud Storage and 
BigQuery clients do not read `STORAGE_EMULATOR_HOST` or a similar variable, and 
the components have no host option; both autowire a client bean (`Storage`, 
`GoogleBigQueryConnectionFactory`), which is how they were pointed at the 
emulators below.
   
   ### Verification
   
   Tested with Camel JBang 4.24.0-SNAPSHOT:
   - Pub/Sub: `gcr.io/google.com/cloudsdktool/google-cloud-cli:emulators` 
(`gcloud beta emulators pubsub start`), topics and subscriptions made with its 
REST API, some with a dead-letter policy (`maxDeliveryAttempts: 5`).
   - Cloud Storage: `fsouza/fake-gcs-server` (memory backend), through an 
autowired `Storage` bean.
   - BigQuery: `ghcr.io/goccy/bigquery-emulator`, through an autowired 
`GoogleBigQueryConnectionFactory` bean.
   - A small logging HTTP proxy in front of fake-gcs-server and the BigQuery 
emulator captured the request bodies.
   - Workload Identity Federation: an `external_account` file with a made-up 
pool and subject token, against the real Google STS (no Google account 
involved).
   
   | Case | Result |
   |---|---|
   | All six Kamelets with the new options unset, `main` and this branch | 
`camel get endpoint --json` shows the same six endpoint URIs |
   | `google-pubsub-source`, one message, unset / `ackMode: AUTO` / 
`synchronousPull` | Received once in 100 s |
   | `ackMode: NONE`, streaming pull | Received once in 100 s: the client keeps 
extending the deadline (`maxAckExtensionPeriod` 3600 s) |
   | `ackMode: NONE`, `maxAckExtensionPeriod: 5` | Received again every ~5 s 
(22 times in 100 s) |
   | `ackMode: NONE`, `synchronousPull: true`, subscription deadline 10 s | 
Received again every 10 s (11 times in 100 s) |
   | `ackMode: NONE`, route calls 
`${header.CamelGooglePubsubAcknowledge.ack(${exchange})}` (Simple) or 
`request.headers.CamelGooglePubsubAcknowledge.ack(exchange)` (Groovy) | 
Received once, not again after the 10 s deadline |
   | Failing route, dead-letter subscription (max 5), `maxDeliveryAttempts` 
unset | `Auto-fetched maxDeliveryAttempts=5`; attempts 1–4 processed 
(`CamelGooglePubsubDeliveryAttempt` 1–4), attempt 5 nacked unprocessed; message 
on the dead-letter topic |
   | The same with `maxDeliveryAttempts: 2` | Attempt 1 processed, attempts 2–5 
nacked unprocessed; still moved to the dead-letter topic after 5 (the 
subscription's limit) |
   | The same with `maxDeliveryAttempts: 0` | All 5 attempts processed |
   | Subscription without a dead-letter policy | No 
`CamelGooglePubsubDeliveryAttempt` header |
   | `maxDeliveryAttempts` unset, policy cannot be read (subscription missing) 
| WARN `Failed to auto-fetch maxDeliveryAttempts ... enforcement will be 
disabled` |
   | `google-storage-event-based-source`, `ackMode` unset / `NONE` + 
`synchronousPull` | Notification received once / again after 10 s |
   | The same, failing route, `maxDeliveryAttempts: 2` | Attempt 1 processed, 
attempts 2–5 nacked unprocessed |
   | `google-pubsub-sink`, `CamelGooglePubsubOrderingKey` set, options unset | 
Every message fails: `Cannot publish a message with an ordering key when 
message ordering is not enabled` |
   | `messageOrderingEnabled: true` | Published; WARN `In conjunction with 
enabling message ordering the pubsubEndpoint should be set` |
   | `messageOrderingEnabled: true`, `pubsubEndpoint: localhost:<emulator>` | 
Published, no warning. Ten messages 20 ms apart arrive on an ordered 
subscription in publish order |
   | `pubsubEndpoint: localhost:<emulator>` without the component's emulator 
`endpoint` | Published to the emulator (message IDs from its sequence): the 
option alone chooses the service |
   | `google-storage-source`, `moveAfterRead: true`, `destinationBucket: 
archive` | `a.txt` gone from `inbox`, present in `archive` |
   | The same with `deleteAfterRead: false` | Copied to `archive`, kept in the 
source bucket |
   | `moveAfterRead`, failing route | Not moved, not deleted; read again |
   | `moveAfterRead`, destination bucket missing | WARN `Error occurred during 
moving or deleting object` (404); file kept, read again every poll |
   | `includeBody: false`, before the template step | Every file fails: `No 
type converter available to convert from type: com.google.cloud.storage.Blob to 
the required type: byte[]`; read again every 500 ms |
   | `includeBody: false`, with it | Body empty, 
`CamelGoogleCloudStorageContentLength: 7`; file deleted |
   | `includeBody: false` + `moveAfterRead` | Moved without downloading; body 
empty |
   | `includeBody: true` | Body is the content, as when unset |
   | `includeFolders` unset / `false`, bucket with `docs/` and `docs/x.txt` | 
Both read / only `docs/x.txt` |
   | `deleteAfterRead: false`, `delay` unset / `5000` | Same file read every 
~500 ms / every 5 s |
   | `google-storage-sink`, `objectName: "status/latest 100%+x.json"`, `file` 
header set | Both messages written to `status/latest 100%+x.json`, as written |
   | `autoCreateBucket: true`, `storageLocation: europe-west1` / unset | 
Captured create request: `"location":"europe-west1"` / `"location":"US-EAST1"` 
(both `"storageClass":"STANDARD"`; the upload request sets no class) |
   | `storageClass: NEARLINE` (option tried, then left out) | Route fails at 
start: no type converter from String to `StorageClass` |
   | Source and sink, `useWorkloadIdentityFederation: true`, 
`workloadIdentityConfig: file:cred.json` | Google STS answers `invalid_target` 
for the made-up pool: the file is loaded and the token exchange attempted. A 
local `token_url` is refused by the Google client (`The provided token URL is 
invalid`) |
   | The same plus `impersonatedServiceAccount` | Fails with `Unable to refresh 
sourceCredentials` from the impersonating credentials, caused by the same STS 
answer |
   | `google-bigquery-sink`, `useAsInsertId: orderId`, same row twice | 
Captured: `{"rows":[{"insertId":"o-1","json":{...}}]}` both times |
   | `useAsInsertId` unset | Captured: `{"rows":[{"json":{...}}]}`, no 
`insertId` |
   | JSON array of two rows | Each row has its own `insertId` (`o-3`, `o-4`) |
   | Row without the field | Sent without `insertId` |
   | `useAsInsertId: num` (a number field) | Message fails with 
`ClassCastException: Integer cannot be cast to String`; nothing sent |
   
   The BigQuery emulator stores every row, also a repeated insert ID: the 
de-duplication is done by BigQuery on a best-effort basis, so only the request 
was checked, not the de-duplication itself.
   
   `./mvnw clean verify -DskipTests` and then `./mvnw clean verify` passed from 
the root, the latter after the header counts of `KameletsCatalogTest` were 
updated. Before building, the library copies of the six Kamelets were refreshed 
with `./mvnw -pl library/camel-kamelets 
resources:copy-resources@copy-resource-one`. The build regenerated 
`kamelets-index.json`.
   
   _Claude Code on behalf of Claus Ibsen_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to