davsclaus opened a new pull request, #3233:
URL: https://github.com/apache/camel-kamelets/pull/3233

   Part of #3170: the Kinesis and DynamoDB Streams sources can read fewer 
records in one poll, and the Kinesis source can read one shard; the CloudWatch 
sink can take the metric name, value and unit as options instead of headers; 
the CloudTrail source can tune how it looks back for events that CloudTrail 
made visible late.
   
   ### The options
   
   | Kamelet | Option | Template mapping | What it does |
   |---|---|---|---|
   | `aws-kinesis-source` | `maxResultsPerRequest` | `maxResultsPerRequest: 
"{{?maxResultsPerRequest}}"` | The most records read from each shard in one 
poll (1000 when unset). |
   | `aws-kinesis-source` | `shardId` | `shardId: "{{?shardId}}"` | Reads one 
shard only, such as `shardId-000000000001`. `pattern: ^shardId-[0-9]+$`. |
   | `aws-kinesis-source` | `shardClosed` | `shardClosed: "{{?shardClosed}}"` | 
`ignore` (component default), `silent` or `fail`, once a shard closed by 
resharding has been read to its end. |
   | `aws-kinesis-source` | `applicationName` | `applicationName: 
"{{?applicationName}}"` | The KCL application, which names the DynamoDB table 
of the KCL (the stream name when unset). Only with `useKclConsumers`. `pattern: 
^[A-Za-z0-9_.-]{3,255}$`, the rules of a DynamoDB table name. |
   | `aws-cloudwatch-sink` | `metricName` | `name: "RAW({{?metricName}})"` | 
The metric name for the messages without a `metric-name` header. `pattern: ^[ 
-~]{1,255}$`. |
   | `aws-cloudwatch-sink` | `metricValue` | `value: "{{?metricValue}}"` | The 
value for the messages without `metric-value` (1 when neither is set). |
   | `aws-cloudwatch-sink` | `metricUnit` | `unit: "{{?metricUnit}}"` | The 
unit for the messages without `metric-unit` (`Count` when neither is set); an 
`enum` of the 27 CloudWatch units. |
   | `aws-cloudtrail-source` | `lookBackWindow` | `lookBackWindow: 
"{{?lookBackWindow}}"` | How far back, in ms, it looks up the events again 
(600000 when unset); 0 turns it off. |
   | `aws-cloudtrail-source` | `lookBackInterval` | `lookBackInterval: 
"{{?lookBackInterval}}"` | How often, in ms, it does so (60000 when unset). |
   | `aws-ddb-streams-source` | `maxResultsPerRequest` | `maxResultsPerRequest: 
"{{?maxResultsPerRequest}}"` | The most changes read in one poll, shared 
between the shards (100 when unset). |
   
   - **Defaults:** all the options are optional and have no `default`, as in 
#3174. When they are unset, the optional placeholders drop them, and the 
endpoint URIs are byte-identical to `main`'s.
   - **Option names:** the CloudWatch options are `metricName`, `metricValue` 
and `metricUnit`, as `name`, `value` and `unit` say too little next to 
`cwNamespace`. None of the names is one the `kamelet:` component takes (#3201).
   - **A header wins over the option** in the CloudWatch sink, as the component 
does: the template sets `CamelAwsCwMetricName` and the others only from the 
headers, and the component falls back on the endpoint options.
   - **`RAW()` on `metricName`:** without it, `Orders Processed+1%` arrived in 
CloudWatch as `Orders Processed 1%`. With it, `Orders Processed+1% (eu)` is 
kept as written.
   - **`lookBackWindow` and `lookBackInterval` need Camel 4.24:** they are not 
in the released `camel-aws-cloudtrail` 4.22.0 and 4.23.0 jars. The descriptions 
say "Needs Camel 4.24 or later".
   - **`sequenceNumber` and `messageTimestamp`** of `aws-kinesis-source` came 
with #3172; this PR tests them again, below.
   - **Not added:**
     - `aws-ddb-sink` `consistentRead`: the component uses it only for 
`GetItem`, `Query`, `Scan` and `ExecuteStatement`. The JSON transformation of 
the sink supports `PutItem`, `UpdateItem` and `DeleteItem` only: a message with 
`CamelAwsDdbOperation: GetItem` fails with `Unsupported operation 'GetItem'`. 
Read operations would also need an output that a sink in a Pipe does not have, 
so neither the read operations nor `consistentRead` are added.
   - **Descriptions:**
     - `aws-kinesis-source` "Good to know": it reads all the shards, up to 1000 
records from each in one poll; it goes on with the new shards after a reshard, 
but stops reading with `shardClosed: fail`; the KCL table is named after 
`applicationName`, or after the stream.
     - `shardClosed: fail` is documented as tested: once a shard is closed, 
every later poll fails, and no records are read from any shard, also not from 
the new ones.
     - `aws-cloudwatch-sink`: the summary, Input and "Good to know" say the 
options are the fallback for the headers, and that an unknown unit fails the 
message.
     - `aws-cloudtrail-source` "Good to know" was wrong: it said the Kamelet 
also reads the events from before it starts, and reads them again after a 
restart. Since Camel 4.22 (`doStart` sets the cursor to the start time; checked 
in the 4.22.0 and 4.23.0 jars and in a run) it reads only the events after it 
starts, and the events that happen while it is stopped are not read. It now 
says so, and explains the late-event lookup.
     - `aws-ddb-streams-source` "Good to know": up to 100 changes in one poll.
   - **Partial pages:** new sections "One shard, and smaller polls" and 
`applicationName` in the KCL example (Kinesis), "The Same Metric for Every 
Message" (CloudWatch), "Events That Become Visible Late" (CloudTrail), "Fewer 
Changes in One Poll" (DynamoDB Streams).
   
   ### Verification
   
   Tested with Camel JBang 4.24.0-SNAPSHOT against `floci/floci` 2.5.0, which 
has Kinesis (with `SplitShard`), CloudWatch, CloudTrail, DynamoDB and DynamoDB 
Streams. Floci's CloudTrail returns no events, so the look-back cases ran 
against a small fake `LookupEvents` server that logs each request and can add 
an event with an older event time. Each case ran with `camel run 
--local-kamelet-dir`, and `camel get endpoint --json` gave the endpoint URI.
   
   **Options unset**
   
   | Case | Result |
   |---|---|
   | All five Kamelets (`aws-ddb-sink` too), the same parameters, `main` and 
this branch | Byte-identical endpoint URIs |
   
   **`aws-kinesis-source`**, stream `w6s` with records `r1`…`r5` (`r1`–`r3`, 
then a 4 s gap, then `r4`, `r5`), and stream `w6split`, whose shard 0 was split 
into shards 1 and 2 before six records were written
   
   | Case | Result |
   |---|---|
   | Unset | `r1`…`r5` |
   | `AT_SEQUENCE_NUMBER`, the sequence number of `r2` | `r2`…`r5` |
   | `AFTER_SEQUENCE_NUMBER`, the same | `r3`…`r5` |
   | `AT_TIMESTAMP`, a time in the gap | `r4`, `r5` |
   | `delay: 3000` | All five in the first poll |
   | `maxResultsPerRequest: 2`, `delay: 3000` | `r1`,`r2` at :29.1, `r3`,`r4` 
at :32.1, `r5` at :35.2: two per poll; the URI has `maxResultsPerRequest=2` |
   | `w6split`, unset | The six records of shards 1 and 2, then `WARN The shard 
with id=shardId-000000000000 ... reached CLOSE status`, once |
   | `shardId: shardId-000000000001` | Only the four records of shard 1 |
   | `shardId: shardId-000000000000` (the closed shard) | Nothing, and the 
CLOSE warning |
   | `shardId: shardId-000000000009` | Nothing; `WARN Configured shardId ... 
not found in shard list` on every poll |
   | `shardClosed: ignore`, four records written 7 s after the start | The six 
records, the CLOSE warning once, then the four new records |
   | `shardClosed: silent` | The six records, no warning |
   | `shardClosed: fail`, four records written 7 s after the start | The six 
records in the first poll; then every poll fails with 
`ReachedClosedStatusException`, and the four new records are not read |
   | `shardId` of the closed shard + `fail` | Every poll fails with 
`ReachedClosedStatusException` |
   | `useKclConsumers` + `applicationName: w6-orders-app`, and a second 
integration without it (`AWS_ENDPOINT_URL` pointed the KCL's own DynamoDB 
client at Floci) | DynamoDB tables `w6-orders-app` and `w6s` were created, 
holding the KCL's worker and leader entries. Floci's KCL support stops there: 
no shard leases, so no record was read in 150 s |
   
   **`aws-cloudwatch-sink`**, data points checked with `aws cloudwatch 
list-metrics` and `get-metric-statistics`
   
   | Case | Result |
   |---|---|
   | Headers `metric-name: Hdr1`, `metric-value: 5`, `metric-unit: Seconds`, 
`main` and this branch | `Hdr1`, 5, `Seconds`, as before |
   | No headers; `metricName: OrdersProcessed`, `metricValue: 42.5`, 
`metricUnit: Milliseconds` | `OrdersProcessed`, 42.5, `Milliseconds`; URI 
`name=OrdersProcessed&unit=Milliseconds&value=42.5` |
   | Headers `FromHeader`/7/`Bytes` and options `FromOption`/99/`Seconds` | 
`FromHeader`, 7, `Bytes`: the header wins |
   | Only `metricName: Ticks` | `Ticks`, 1, `Count` |
   | `metricName: Rate`, `metricValue: 2048`, `metricUnit: Bytes/Second` | 
`Rate`, 2048, `Bytes/Second` |
   | `metricName: "Orders Processed+1% (eu)"` | Kept as written (without 
`RAW()`: `Orders Processed 1%`) |
   | No header and no option, `main` and this branch | Fails: `The parameter 
MetricData.member.1.MetricName is required` |
   | Header `metric-unit: Parsecs` | Fails: `Unknown unit Parsecs: use one of 
Seconds, ...` |
   
   **`aws-cloudtrail-source`**, against the fake: an event `OLD` from one hour 
before the start; 20 s after the start the event `NEW1`, then 2 s later `LATE`, 
with an event time 5 s ago, so older than `NEW1`
   
   | Case | Result |
   |---|---|
   | Unset, `main` and this branch | `NEW1` at once; `LATE` 60 s after the 
start, from a lookup with `EndTime`; `OLD` never |
   | `lookBackWindow: 0` | `NEW1` only; no lookup with `EndTime` |
   | `lookBackInterval: 5000` | `NEW1`, then `LATE` 2 s after it was added |
   | `lookBackWindow: 3000`, `lookBackInterval: 5000` | `NEW1` only: `LATE` is 
older than the window |
   | `lookBackInterval: 0` | Does not start: `lookBackInterval must be above 0 
unless lookBackWindow is 0, but was 0` |
   
   **`aws-ddb-streams-source`**, table `w6t` with a stream; five items written 
in one `batch-write-item` 15 s after the start
   
   | Case | Result |
   |---|---|
   | Unset, `main` and this branch | Five `INSERT`s at once |
   | `delay: 3000` | Five at once |
   | `maxResultsPerRequest: 2`, `delay: 3000` | 2, 2 and 1, 3 s apart; the URI 
has `maxResultsPerRequest=2` |
   | `maxResultsPerRequest: 1`, `delay: 3000` | One every 3 s |
   
   **`aws-ddb-sink`**
   
   | Case | Result |
   |---|---|
   | `CamelAwsDdbOperation: GetItem` | Fails in the transformation: 
`Unsupported operation 'GetItem'` |
   
   `./mvnw clean verify -DskipTests` and then `./mvnw clean verify` passed from 
the root. Before building, the library copies of the four Kamelets were 
refreshed with `./mvnw -pl library/camel-kamelets 
resources:copy-resources@copy-resource-one`. The build regenerated 
`kamelets-index.json`.
   
   _Claude Code on behalf of Claus Ibsen_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to