Tzu-Li (Gordon) Tai created FLINK-6349:
------------------------------------------
Summary: Enforce per-subtask record ordering on resharding for
FlinkKinesisConsumer
Key: FLINK-6349
URL: https://issues.apache.org/jira/browse/FLINK-6349
Project: Flink
Issue Type: Improvement
Components: Kinesis Connector
Reporter: Tzu-Li (Gordon) Tai
As described in FLINK-6316, currently the Kinesis consumer does not provide any
ordering guarantees when resharding occurs.
While this cannot be enforced globally (i.e. if a merged / split shard's child
shard ends up in a different subtask, we cannot do any coordination for
ordering guarantee), we can definitely enforce this locally for each subtask.
Simply put, we can still locally enforce ordering by making sure that
discovered child shards are consumed only after any of its parent shards that
were on the same subtask are fully consumed.
To do this, we would also need to add "parent shard" information to
{{KinesisStreamShard}} (Flink's representation of Kinesis shards).
This would be directly beneficial for per-shard watermarks (FLINK-5697) to
retain per-shard time characteristics after a reshard, and therefore can be
seen as a prerequisite.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
