[
https://issues.apache.org/jira/browse/FLINK-5697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672285#comment-16672285
]
ASF GitHub Bot commented on FLINK-5697:
---------------------------------------
EronWright commented on issue #6980: [FLINK-5697] [kinesis] Add periodic
per-shard watermark support
URL: https://github.com/apache/flink/pull/6980#issuecomment-435202809
There is a caveat with this implementation that the docs should perhaps
mention. The caveat is that it may produce spurious late events when
processing a backlog of data.
Here's an example of when that may occur. Imagine that subtask 1 is
processing shard A and subtask 2 is processing shard B. Shard A has reached
6:00 in event time (as per the assigner), and so the subtask emits the
corresponding watermark. At this point, the subtask has made the irrevocable
assertion that subsequent events will be past 6:00. Meanwhile, Shard B is at
5:30 and undergoes a split into C/D. If either shard is subsequently assigned
to subtask 1, the events will be considered late due to the assertion made
earlier.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Add per-shard watermarks for FlinkKinesisConsumer
> -------------------------------------------------
>
> Key: FLINK-5697
> URL: https://issues.apache.org/jira/browse/FLINK-5697
> Project: Flink
> Issue Type: New Feature
> Components: Kinesis Connector, Streaming Connectors
> Reporter: Tzu-Li (Gordon) Tai
> Assignee: Thomas Weise
> Priority: Major
> Labels: pull-request-available
>
> It would be nice to let the Kinesis consumer be on-par in functionality with
> the Kafka consumer, since they share very similar abstractions. Per-partition
> / shard watermarks is something we can add also to the Kinesis consumer.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
