Claus Ibsen created CAMEL-24197:
-----------------------------------
Summary: camel-aws2-kinesis - Fixed-shardId DescribeStream on
every poll and batch partition key
Key: CAMEL-24197
URL: https://issues.apache.org/jira/browse/CAMEL-24197
Project: Camel
Issue Type: Bug
Components: camel-aws
Reporter: Claus Ibsen
Split from CAMEL-24157 (medium-severity findings, group 3).
Two remaining Kinesis findings:
*1. Fixed-shardId consumer calls DescribeStream on every poll*
{{Kinesis2Consumer:108-141}} — When {{shardId}} is configured (non-empty),
every call to {{poll()}} issues a {{DescribeStream}} API call to locate the
Shard object. Two problems:
- Rate-limit risk: DescribeStream has a 10 TPS per-account limit. With default
1-second poll delay, a single consumer approaches the limit; multiple consumers
exceed it.
- >100 shard failure: DescribeStream returns at most 100 shards per response.
The code does not paginate (no {{exclusiveStartShardId}} loop), so for streams
with >100 shards the configured shard may not be found, throwing
{{IllegalStateException}}.
The multi-shard path correctly uses a cached {{ShardMonitor}} with the
{{ListShards}} API (paginated, higher rate limit). The fixed-shardId path
bypasses this caching entirely.
*2. Batch producer applies single partition key to all records*
{{Kinesis2Producer:77-122}} — A single {{CamelAwsKinesisPartitionKey}} header
is applied to every {{PutRecordsRequestEntry}} in the batch. All records are
routed to the same shard. This is a design limitation — no per-record partition
key mechanism exists in the current API.
*Fix approach:*
- #1: Cache the shard iterator for the configured shardId (similar to
ShardMonitor), avoiding DescribeStream on every poll. Add pagination for
>100-shard streams.
- #2: Design decision needed — consider accepting
{{List<PutRecordsRequestEntry>}} directly, or a wrapper type carrying
per-record keys, or deriving partition keys from each record body.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
