[
https://issues.apache.org/jira/browse/FLINK-4616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15863326#comment-15863326
]
ASF GitHub Bot commented on FLINK-4616:
---------------------------------------
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/3031
Hi @MayerRoman! Thank you for coming back to this issue.
I had a quick chat offline with @aljoscha about whether or not it'll be
reasonable to add this. Either your approach in this PR or letting window
operators checkpoint watermarks will both solve the issue of late elements
after restore. We thought that we should let the user function code be free of
responsibility of checkpointing watermarks, and let user code simply leave that
to the streaming internals (checkpointed by window operators, and perhaps also
by source operators).
So, essentially, the Kafka consumer should not need to checkpoint
watermarks, and we can close this PR and the JIRA ticket. Very sorry for the
late discussion on this, and having you worked on it already.
Let we know what you think and whether or not you agree :-)
> Kafka consumer doesn't store last emmited watermarks per partition in state
> ---------------------------------------------------------------------------
>
> Key: FLINK-4616
> URL: https://issues.apache.org/jira/browse/FLINK-4616
> Project: Flink
> Issue Type: Bug
> Components: Kafka Connector
> Affects Versions: 1.1.1
> Reporter: Yuri Makhno
> Assignee: Roman Maier
>
> Kafka consumers stores in state only kafka offsets and doesn't store last
> emmited watermarks, this may go to wrong state when checkpoint is restored:
> Let's say our watermark is (timestamp - 10) and in case we have the following
> messages queue results will be different after checkpoint restore and during
> normal processing:
> A(ts = 30)
> B(ts = 35)
> ------ checkpoint goes here
> C(ts=15) -- this one should be filtered by next time window
> D(ts=60)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
