[
https://issues.apache.org/jira/browse/BEAM-11014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Usamah Jassat updated BEAM-11014:
---------------------------------
Description:
There is a bug in the Kinesis connector which doesn't allow Beam to read from
new shards if restored from an old enough checkpoint.
When loading from an old checkpoint `ShardReadersPool` will identify that the
old shard is closed and only attempts to read from successive shards. However
when trying to find successive shards it will only look for shards that were
children of the old shard, these shards may not exist if the checkpoint is old
enough as expired shards are not returned when listing shards
([docs|https://docs.aws.amazon.com/streams/latest/dev/kinesis-using-sdk-java-after-resharding.html])
Example 1:
Checkpoint:
(Alive)
Shard 1
At restoration:
(Alive)
> Shard 2 \
(Exp) / \ (Alive)
Shard 1 -> Shard 4
\ (Alive) /
> Shard 3 /
In this example the connector will currently work correctly as Shard 2 and 3
will be identified as the successive shards and will continue reading from them.
Example 2:
Checkpoint:
(Alive)
Shard 1
At restoration:
(Exp)
> Shard 2 \
(Exp) / \ (Alive)
Shard 1 -> Shard 4
\ (Exp) /
> Shard 3 /
In this example the connector currently won't work correctly as it wont
identify Shard 4 as a successive shard as its not a child of Shard 1 and thus
stop reading from the stream when it should start reading from Shard 4.
was:
There is a bug in the Kinesis connector which doesn't allow Beam to read from
new shards if restored from an old enough checkpoint.
When loading from an old checkpoint `ShardReadersPool` will identify that the
old shard is closed and only attempts to read from successive shards. However
when trying to find successive shards it will only look for shards that were
children of the old shard, these shards may not exist if the checkpoint is old
enough as expired shards are not returned when listing shards
([docs|https://docs.aws.amazon.com/streams/latest/dev/kinesis-using-sdk-java-after-resharding.html])
Example 1:
Checkpoint:
(Alive)
Shard 1
At restoration:
(Alive)
> Shard 2 \
(Exp) / \ (Alive)
Shard 1 -> Shard 4
\ (Alive) /
> Shard 3 /
In this example the connector will currently work correctly as Shard 2 and 3
will be identified as the successive shards and will continue reading from them.
Example 2:
Checkpoint:
(Alive)
Shard 1
At restoration:
(Exp)
> Shard 2 \
(Exp) / \ (Alive)
Shard 1 -> Shard 4
\ (Exp) /
> Shard 3 /
In this example the connector currently won't work correctly as it wont
identify Shard 4 as a successive shard as its not a child of Shard 1 and thus
stop reading from the stream when it should start reading from Shard 4.
> Bug in recovering from checkpoints
> ----------------------------------
>
> Key: BEAM-11014
> URL: https://issues.apache.org/jira/browse/BEAM-11014
> Project: Beam
> Issue Type: Bug
> Components: io-java-kinesis
> Reporter: Usamah Jassat
> Priority: P2
>
> There is a bug in the Kinesis connector which doesn't allow Beam to read from
> new shards if restored from an old enough checkpoint.
> When loading from an old checkpoint `ShardReadersPool` will identify that the
> old shard is closed and only attempts to read from successive shards. However
> when trying to find successive shards it will only look for shards that were
> children of the old shard, these shards may not exist if the checkpoint is
> old enough as expired shards are not returned when listing shards
> ([docs|https://docs.aws.amazon.com/streams/latest/dev/kinesis-using-sdk-java-after-resharding.html])
> Example 1:
> Checkpoint:
> (Alive)
> Shard 1
> At restoration:
> (Alive)
> > Shard 2 \
> (Exp) / \ (Alive)
> Shard 1 -> Shard 4
> \ (Alive) /
> > Shard 3 /
> In this example the connector will currently work correctly as Shard 2 and 3
> will be identified as the successive shards and will continue reading from
> them.
> Example 2:
> Checkpoint:
> (Alive)
> Shard 1
> At restoration:
> (Exp)
> > Shard 2 \
> (Exp) / \ (Alive)
> Shard 1 -> Shard 4
> \ (Exp) /
> > Shard 3 /
> In this example the connector currently won't work correctly as it wont
> identify Shard 4 as a successive shard as its not a child of Shard 1 and thus
> stop reading from the stream when it should start reading from Shard 4.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
