Claus Ibsen created CAMEL-27180:
-----------------------------------

             Summary: camel-aws2-kinesis - shardClosed=fail fails every poll 
after a reshard, forever
                 Key: CAMEL-27180
                 URL: https://issues.apache.org/jira/browse/CAMEL-27180
             Project: Camel
          Issue Type: Bug
          Components: camel-aws2-kinesis
            Reporter: Claus Ibsen


h3. Problem
The polling {{aws2-kinesis}} consumer without {{shardId}} reads all the shards 
of the stream. After a reshard ({{SplitShard}} or {{MergeShards}}) the parent 
shard is closed, but it stays in the {{ListShards}} result for the retention 
period of the stream (24 hours by default, up to a year). Since CAMEL-25569 the 
consumer goes on with the child shards.

With {{shardClosed=fail}}, once the parent shard has been read to its end, 
*every* later poll fails: {{handleClosedShard}} throws 
{{IllegalStateException(ReachedClosedStatusException)}} inside the 
{{parallelStream().forEach(...)}} of {{poll()}}, and the exception leaves 
{{poll()}}. But the child shards are still read, as the other tasks of the 
parallel stream keep running. So the option neither stops the consumer nor 
reports the closed shard once; it logs a WARN with a full stack trace on every 
poll until the closed shard leaves the shard list. Also:
* {{ForkJoinTask.invoke()}} returns on the first exception, so the next poll 
can start while the shard tasks of the failed poll are still reading and 
routing.
* the number of routed records of the poll is lost, and the 
{{forceConsumerAsReady()}} at the end of {{poll()}} is skipped.

The option description is also out of date: it says that with {{ignore}} and 
{{silent}} "the consumer will not process new messages until restarted", which 
is only true with {{shardId}} set (without it, the child shards are read), and 
it names the exception {{ReachedClosedStateException}} instead of 
{{ReachedClosedStatusException}}.

h3. How to reproduce
Stream {{w6split}} with one shard, a few records, then {{aws kinesis 
split-shard}} into shards 1 and 2 (tested with Camel 4.24.0-SNAPSHOT against a 
local Kinesis emulator):
{noformat}
aws2-kinesis://w6split?iteratorType=TRIM_HORIZON&shardClosed=fail&delay=500&...
{noformat}
The first poll routes the records of all three shards. From then on, every 500 
ms:
{noformat}
WARN  Kinesis2Consumer : Failed polling endpoint: 
aws2-kinesis://w6split?...&shardClosed=fail...
java.lang.IllegalStateException: java.lang.IllegalStateException: 
org.apache.camel.component.aws2.kinesis.ReachedClosedStatusException
        ...
        at 
org.apache.camel.component.aws2.kinesis.Kinesis2Consumer.poll(Kinesis2Consumer.java:146)
Caused by: java.lang.IllegalStateException: 
org.apache.camel.component.aws2.kinesis.ReachedClosedStatusException
        at 
org.apache.camel.component.aws2.kinesis.Kinesis2Consumer.handleClosedShard(Kinesis2Consumer.java:365)
        at 
org.apache.camel.component.aws2.kinesis.Kinesis2Consumer.getShardIterator(Kinesis2Consumer.java:279)
{noformat}
29 such failures in 15 seconds; records written to the child shards in that 
time were still routed.

h3. Where
* 
{{components/camel-aws/camel-aws2-kinesis/src/main/java/org/apache/camel/component/aws2/kinesis/Kinesis2Consumer.java}}:
 {{poll()}} (the {{parallelStream().forEach}} over {{getCurrentShardList()}}), 
{{getShardIterator(...)}} (calls {{handleClosedShard}} on every poll for a 
closed shard), {{handleClosedShard(...)}} ({{case fail}}).
* 
{{components/camel-aws/camel-aws2-kinesis/src/main/java/org/apache/camel/component/aws2/kinesis/Kinesis2Configuration.java}}:
 the description of {{shardClosed}}.
* The only test, {{KinesisConsumerClosedShardWithFailTest}}, has a single shard.

h3. Suggested change
Decide what {{fail}} means when all the shards are read. For example:
* report the {{ReachedClosedStatusException}} once per closed shard (through 
the consumer's {{ExceptionHandler}}, so {{bridgeErrorHandler}} works), and then 
skip that shard like {{silent}}, while the other shards go on; or
* keep the current meaning (stop with an error) only with {{shardId}} set, and 
treat a closed parent shard with known children as normal resharding.
In either case, do not throw out of the parallel stream while other shard tasks 
run (see also CAMEL-26670, which moves the shard reads to a Camel-managed 
executor), fix the option description, and add a multi-shard test.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to