[
https://issues.apache.org/jira/browse/BEAM-11014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17277323#comment-17277323
]
Beam JIRA Bot commented on BEAM-11014:
--------------------------------------
This issue was marked "stale-P2" and has not received a public comment in 14
days. It is now automatically moved to P3. If you are still affected by it, you
can comment and move it back to P2.
> Bug in recovering from checkpoints
> ----------------------------------
>
> Key: BEAM-11014
> URL: https://issues.apache.org/jira/browse/BEAM-11014
> Project: Beam
> Issue Type: Bug
> Components: io-java-kinesis
> Reporter: Usamah Jassat
> Priority: P3
>
> There is a bug in the Kinesis connector which doesn't allow Beam to read from
> new shards if restored from an old enough checkpoint.
> When loading from an old checkpoint `ShardReadersPool` will identify that the
> old shard is closed and only attempts to read from successive shards. However
> when trying to find successive shards it will only look for shards that were
> children of the old shard, these shards may not exist if the checkpoint is
> old enough as expired shards are not returned when listing shards
> ([docs|https://docs.aws.amazon.com/streams/latest/dev/kinesis-using-sdk-java-after-resharding.html])
> Example 1:
> Checkpoint:
> (Alive)
> Shard 1
> At restoration:
> (Alive)
> > Shard 2 \
> (Exp) / \ (Alive)
> Shard 1 -> Shard 4
> \ (Alive) /
> > Shard 3 /
> In this example the connector will currently work correctly as Shard 2 and 3
> will be identified as the successive shards and will continue reading from
> them.
> Example 2:
> Checkpoint:
> (Alive)
> Shard 1
> At restoration:
> (Exp)
> > Shard 2 \
> (Exp) / \ (Alive)
> Shard 1 -> Shard 4
> \ (Exp) /
> > Shard 3 /
> In this example the connector currently won't work correctly as it wont
> identify Shard 4 as a successive shard as its not a child of Shard 1 and thus
> stop reading from the stream when it should start reading from Shard 4.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
