davsclaus opened a new pull request, #3229:
URL: https://github.com/apache/camel-kamelets/pull/3229

   Part of #3170: the Google Calendar, Gmail and Google Sheets Kamelets can log 
in with a service account, with domain-wide delegation, and take the paging and 
rendering options of their APIs; `google-vertexai-sink` can call partner models 
such as Claude with `rawPredict`, and ask for JSON answers; `ibm-cos-source` 
can read one file, and `ibm-cos-sink` can delete the local file it uploaded.
   
   ### The options
   
   All new options are optional and have no `default`. When they are not set, 
the optional placeholders drop them, and the endpoint URI is the same as on 
`main` (checked with `camel get endpoint`, table below).
   
   | Kamelet | Option | Template mapping | What it does |
   |---|---|---|---|
   | `google-calendar-source` | `serviceAccountKey` | 
`serviceAccountKey=RAW({{?serviceAccountKey}})` | Logs in with the JSON key of 
a service account: `file:`, `classpath:` or `base64:` (Camel resource loading). 
A secret. |
   | | `delegate` | `delegate=RAW({{?delegate}})` | Reads the calendar of this 
Workspace user (domain-wide delegation). |
   | | `scopes` | `scopes={{?scopes}}` | The scopes for the service account, 
such as `calendar.readonly`; the component default is 
`https://www.googleapis.com/auth/calendar`. |
   | | `maxResults` | `maxResults={{?maxResults}}` | Events per page (component 
default 10). |
   | | `query` | `query=RAW({{?query}})` | Free-text search, the `q` of the 
Calendar API. |
   | | `considerLastUpdate` | `considerLastUpdate={{?considerLastUpdate}}` | 
Each poll reads all pages, then only the events changed since the last one 
received. |
   | `google-mail-source` | `serviceAccountKey`, `delegate`, `scopes` | as 
above | Service account login. Here `scopes` is needed: the component has no 
default. |
   | | `maxResults` | `maxResults={{?maxResults}}` | Emails per poll (component 
default 10). |
   | | `raw` | `raw={{?raw}}` | The body is the whole email, with attachments, 
base64url-encoded, as JSON text. |
   | `google-sheets-source` | `serviceAccountKey`, `delegate`, `scopes` | as 
above | Service account login; `scopes` is needed. |
   | | `valueRenderOption` | `valueRenderOption={{?valueRenderOption}}` | 
`FORMATTED_VALUE`, `UNFORMATTED_VALUE` or `FORMULA`, an `enum`. |
   | | `maxResults` | `maxResults={{?maxResults}}` | The most rows of each 
range to send. Camel applies this limit after the read: Google still returns 
the whole range. |
   | `google-sheets-sink` | `serviceAccountKey`, `delegate`, `scopes` | 
`serviceAccountKey: "RAW({{?serviceAccountKey}})"`, `delegate: 
"RAW({{?delegate}})"`, `scopes: "{{?scopes}}"` | Service account login; 
`scopes` is needed. |
   | `google-vertexai-sink` | `publisher` | `publisher: "{{?publisher}}"` | The 
publisher of a partner model for `rawPredict`, such as `anthropic`, `meta` or 
`mistralai`. |
   | | `anthropicVersion` | `anthropicVersion: "{{?anthropicVersion}}"` | The 
`anthropic_version` of requests to Claude (component default 
`vertex-2023-10-16`). |
   | | `jsonMode` | `jsonMode: "{{?jsonMode}}"` | Asks for the response type 
`application/json`. |
   | `ibm-cos-source` | `fileName` | `fileName: "{{?fileName}}"` | Reads only 
the object with this key. |
   | `ibm-cos-sink` | `deleteAfterWrite` | `deleteAfterWrite: 
"{{?deleteAfterWrite}}"` | Deletes the local file of the body once it has been 
uploaded. |
   
   - **The OAuth options are no longer required** in the three Google Workspace 
Kamelets. `clientId`, `clientSecret`, `accessToken` and `refreshToken` went 
from `{{x}}` to `{{?x}}`, so that a service account can be used instead. When 
they are set, the URI is the same as before. With neither kind of login set, 
Calendar and Gmail fail at start with `(clientId and clientSecret) or 
serviceAccountKey are required`. Sheets then uses the Application Default 
Credentials (`GOOGLE_APPLICATION_CREDENTIALS`), which the descriptions now say.
   - **Which login wins:** the components use OAuth when `clientId` and 
`clientSecret` are set (for Gmail, `clientSecret` alone), and the service 
account key only when they are not.
   - **The service account key** is a Camel resource location, as in 
`google-vertexai-sink`: `file:`, `classpath:` or `base64:`. It has a `pattern` 
for these three prefixes, and is mapped in `RAW()`. A `base64:` key that 
contains `+` and `/` was read correctly.
   - **`scopes` with OAuth makes the Kamelet fail.** camel-google-common turns 
the OAuth login into a `GoogleCredential` with service-account scopes, which 
the Google client library refuses with an `IllegalArgumentException`. The 
`scopes` descriptions say to leave it unset with OAuth.
   - **Patterns:**
     - `delegate`: an email address.
     - `scopes`: `https://` URLs, separated by commas.
     - `query` and `fileName`: no control characters.
     - `publisher` and `anthropicVersion`: plain tokens.
   - **Descriptions:**
     - Calendar: the "Good to know" paragraph says that each poll reads one 
page of `maxResults` events, the least recently changed first. Without 
`syncFlow` or `considerLastUpdate`, the same events therefore come again on 
every poll. That is also true on `main`, which reads the first 10.
     - Vertex AI: the `rawPredict` text no longer says that it always fails.
     - The partial pages have new examples: service accounts, raw email, 
Claude, one file, and deleting the local file.
   - **Header:** `google-vertexai-sink` declares the 
`CamelGoogleVertexAIRawResponse` out header. It is a header declaration only, 
with no data type `default`.
   
   #### Left out
   
   - **`candidateCount`** (`google-vertexai-sink`). With 2, the request asks 
for two answers, but the component puts only the first answer in the body: the 
Gen AI SDK logs `This response has 2 candidates, will only use the first 
candidate`. The other answers are paid for and then dropped, so the option is 
useless in a sink until camel-google-vertexai returns all the candidates.
   - **Streaming** (`generateChatStreaming`, `streamRawPredict`, 
`streamOutputMode`). In a sink, the component waits for the whole stream and 
puts the joined text in the body, which is what `generateText` and `rawPredict` 
already give. The `chunks` mode gives a List of the pieces after the stream has 
ended, so nothing arrives earlier.
   
   ### Verification
   
   Camel JBang 4.24.0-SNAPSHOT, `camel run 
--local-kamelet-dir=<worktree>/kamelets`.
   
   **Nothing was tested against the real Google APIs or IBM Cloud.** No Google 
API here has a local emulator, so the requests went to a small local HTTP 
server, a stand-in that records each request and answers like the API. The 
clients and logins were the components' own:
   
   - **Calendar, Gmail and Sheets:** a test `clientFactory` that calls the 
components' own `Batch*ClientFactory`. So the OAuth or service-account 
credential is built by camel-google-common exactly as in production. The test 
factory only rebuilds the client with the root URL of the local server. The 
service account key is a generated RSA key whose `token_uri` points to the 
local server, so the token request, a signed JWT whose `iss`, `sub` (the 
delegate) and `scope` claims were decoded, was recorded too.
   - **Vertex AI, Gemini operations:** the Gen AI SDK's 
`GOOGLE_VERTEX_BASE_URL` environment variable pointed to the local server, with 
the same kind of service account key.
   - **Vertex AI, `rawPredict`:** two ways. First, a key whose token endpoint 
fails, with DEBUG logging of the producer, which logs the endpoint name and the 
request body; the call then failed locally while getting a token, so nothing 
left the machine. Second, a fake `PredictionServiceClient` (autowired) that 
records the request and answers like Claude, to check the body afterwards.
   - **IBM COS:** the Kamelet as it is, with its `apiKey`. The IBM SDK's IAM 
token endpoint (`SDKGlobalConfiguration.IAM_ENDPOINT`) pointed to the local 
server, and `endpointUrl` to `floci/floci`. floci is an S3 stand-in that 
accepts the IBM bearer token. MinIO was not used, because it refuses bearer 
tokens.
   
   | Case | Result |
   |---|---|
   | New options unset, all 7 Kamelets, `main` and this branch, OAuth options 
set where `main` requires them | `camel get endpoint`: identical URIs for all 7 
|
   | Calendar, `serviceAccountKey: file:…`, `delegate: [email protected]`, 
`maxResults: 2`, `query: "team & lunch + 100%"` | Token JWT `sub` = 
`[email protected]`, `scope` = `…/auth/calendar`; `events.list` with 
`maxResults=2`, `q=team & lunch + 100%` (kept as written), `Bearer` token of 
the service account |
   | Calendar, `serviceAccountKey: base64:…`, `maxResults: 2`, 
`considerLastUpdate: true`, 5 events | Poll 1 reads 3 pages (`pageToken` 2, 4), 
5 events; later polls send `updatedMin` = newest update seen; after one event 
changed, only that event came again |
   | Calendar, `maxResults: 2`, `considerLastUpdate` unset | One page per poll; 
the oldest event came again on every poll |
   | Calendar, `scopes: …/calendar.readonly` | JWT `scope` = 
`…/calendar.readonly`; default `maxResults=10` sent |
   | Gmail, OAuth, `maxResults: 2` | `messages.list` with `maxResults=2`, 
`q=is:unread`; 2 emails, `format=FULL`, subject and sender headers set, then 
marked as read |
   | Gmail, `serviceAccountKey`, `delegate: [email protected]`, `scopes: 
…/gmail.modify`, `raw: true`, `maxResults: 1` | JWT `sub` and `scope` as set; 
`messages.get` with `format=RAW`; body a JSON string with the base64url of the 
whole email (attachment included); no subject or sender header |
   | Gmail, `serviceAccountKey` without `scopes` | Token request with an empty 
`scope` claim |
   | Gmail, no login options | Fails at start: `(clientId and clientSecret) or 
serviceAccountKey are required to create Gmail client` |
   | Gmail and Sheets, OAuth with `scopes` set | Fails: 
`IllegalArgumentException` from `GoogleCredential` |
   | Sheets source, OAuth, `valueRenderOption: UNFORMATTED_VALUE`, `maxResults: 
2` | `batchGet` with `valueRenderOption=UNFORMATTED_VALUE` (no `maxResults` 
sent); 2 rows sent of 6, numbers as numbers (`["Apple",1.5,3.0]`) |
   | Sheets source, `serviceAccountKey: base64:…`, `delegate`, `scopes: 
…/spreadsheets.readonly`, `valueRenderOption: FORMULA`, `splitResults: false`, 
`maxResults: 3` | JWT as set; one message with 3 rows and formulas (`"=B2*2"`) |
   | Sheets source, no login options, `scopes`, 
`GOOGLE_APPLICATION_CREDENTIALS` set | Logged in with the Application Default 
Credentials; default `valueRenderOption=FORMATTED_VALUE` sent |
   | Sheets sink, OAuth (as on `main`) | `values:append` with 
`valueInputOption=USER_ENTERED` |
   | Sheets sink, `serviceAccountKey`, `delegate: [email protected]`, 
`scopes` | JWT `sub` = `[email protected]`; row appended with the service 
account token |
   | Sheets sink, no login options, `scopes`, `GOOGLE_APPLICATION_CREDENTIALS` 
set | Appended with the Application Default Credentials |
   | Sheets source and sink, Gmail: `base64:` key containing `+` and `/` | Read 
correctly (`RAW()`) |
   | Vertex AI, `generateText`, unset | `generationConfig` without 
`responseMimeType`; body `Paris (candidate 1)` |
   | Vertex AI, `jsonMode: true` | 
`generationConfig.responseMimeType=application/json`; body JSON text |
   | Vertex AI, `candidateCount: 2` (before it was left out) | Request with 
`candidateCount: 2`; body only the first answer, SDK warning (see "Left out") |
   | Vertex AI, `rawPredict`, `publisher` unset | Fails: `Publisher must be 
specified for rawPredict operation` (as on `main`) |
   | Vertex AI, `rawPredict`, `publisher: anthropic`, text prompt | Endpoint 
`projects/test-project/locations/us-east5/publishers/anthropic/models/claude-sonnet-4-5`;
 request `anthropic_version: vertex-2023-10-16`, `max_tokens: 1024`, 
`temperature`, `top_p`, `top_k`, one user message; body afterwards the text of 
the answer, `CamelGoogleVertexAIRawResponse` the JSON |
   | The same, `anthropicVersion: vertex-2099-01-01` | Request 
`anthropic_version: vertex-2099-01-01` |
   | `rawPredict`, body a JSON Messages request, with and without 
`anthropic_version` | Sent as it is, `anthropic_version` added only when 
missing |
   | `rawPredict`, `publisher: mistralai` | Endpoint 
`…/publishers/mistralai/…`, Mistral request format; body the JSON answer |
   | IBM COS source, `fileName: report.csv`, `deleteAfterRead: false`, 3 files 
in the bucket | Logged in with the API key (IAM request `apikey=…`); only 
`report.csv` read, again on every poll |
   | `fileName: report.csv`, `deleteAfterRead` default | Read once and deleted; 
each later poll logs `NoSuchKey`; uploaded again, read once more; `a.txt`, 
`b.txt` untouched |
   | `fileName: a.txt` with `prefix: zzz/` | `a.txt` read; `prefix` is not used 
|
   | IBM COS sink, `deleteAfterWrite: true`, body from `file-watch-source` (a 
`File`) | Uploaded, local file deleted |
   | The same, body from the `file` component (`GenericFile`), `noop=true` | 
Uploaded, local file deleted |
   | The same, `file` component with its default move to `.camel` | Uploaded 
and deleted, then the file component logs `Cannot rename file` (said in the 
description) |
   | `deleteAfterWrite` unset, `file` component | Uploaded; the local file is 
kept (moved to `.camel`) |
   | `deleteAfterWrite: true`, text body | Uploaded, nothing deleted |
   | The partial-page example: `file-watch-source` → `insert-header-action` 
(`file` = `${header.CamelFileName}`) → sink | `invoice-42.txt` uploaded under 
its name, local file deleted |
   
   #### Found along the way (not changed here)
   
   - **`google-calendar-source` cannot log in with OAuth, on `main` too.** The 
calendar stream component always has scopes (default `…/auth/calendar`). 
camel-google-common's `GoogleCredentialsHelper.createOAuthCredential` passes 
them to `GoogleCredential.Builder.setServiceAccountScopes`, and the Google 
client library refuses scopes on a credential without a private key. So every 
poll fails with `Could not create Google Calendar client`, caused by an 
`IllegalArgumentException`. This needs a fix in camel-google-common. Until 
then, the service account login added here is the only one that works for 
Calendar.
   - **`google-vertexai-sink` does not start in Camel JBang.** On `main` and 
this branch, endpoint start fails with `NoClassDefFoundError: 
com/google/protobuf/RuntimeVersion$RuntimeDomain`, an older protobuf-java on 
the classpath. All the Vertex AI runs above added 
`--dep=com.google.protobuf:protobuf-java:4.35.1`.
   
   ### Build
   
   `./mvnw clean verify -DskipTests` and then `./mvnw clean verify` passed from 
the root. Before building, the library copies of the seven Kamelets were 
refreshed with `./mvnw -pl library/camel-kamelets 
resources:copy-resources@copy-resource-one`. The build regenerated 
`kamelets-index.json`.
   
   _Claude Code on behalf of Claus Ibsen_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to