[
https://issues.apache.org/jira/browse/BEAM-2571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098293#comment-16098293
]
ASF GitHub Bot commented on BEAM-2571:
--------------------------------------
GitHub user aljoscha opened a pull request:
https://github.com/apache/beam/pull/3625
[BEAM-2571] Fix FlinkRunner Watermark issues on 2.1.0 branch
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aljoscha/beam
jira-2571-fix-flink-watermark-issues-2.1.0
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/3625.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3625
----
commit 5c4a95aa63da23027b619a50928ebebe5beb05c2
Author: Aljoscha Krettek <[email protected]>
Date: 2017-07-12T12:39:58Z
[BEAM-2571] Clarify pushedback variable name in DoFnOperator
commit 1686805d84316f2a8438c642214cdabe5b579381
Author: Aljoscha Krettek <[email protected]>
Date: 2017-07-12T12:42:37Z
[BEAM-2571] Change DoFnOperator to use Long.MAX_VALUE as max watermark
This is in line with what Flink does and what BoundedSourceWrapper and
UnboundedSourceWrapper do.
commit ade506526b4ff56eb4ed15e9eea04d1d3345bc13
Author: Aljoscha Krettek <[email protected]>
Date: 2017-07-12T13:38:06Z
[BEAM-2571] Respect watermark contract in Flink DoFnOperator
In Flink, a watermark T specifies that there will be no elements with a
timestamp <= T in the future. In Beam, a watermark T specifies that
there will not be element with a timestamp < T in the future. This leads
to problems when the watermark is exactly "on the timer timestamp", most
prominently, this happened with Triggers, where Flink would fire the
Trigger too early and the Trigger would determine (based on the
watermark) that it is not yet time to fire the window while Flink
thought it was time.
This also adds a test that specifially tests the edge case.
----
> Flink ValidatesRunner failing CombineTest.testSlidingWindowsCombineWithContext
> ------------------------------------------------------------------------------
>
> Key: BEAM-2571
> URL: https://issues.apache.org/jira/browse/BEAM-2571
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
> Reporter: Kenneth Knowles
> Assignee: Aljoscha Krettek
> Fix For: 2.1.0, 2.2.0
>
>
> This appears to have been caused by https://github.com/apache/beam/pull/3429
> which fixes a couple errors in how trigger timers were processed / final
> panes labeled.
> I am investigating, considering roll back vs forward fix. Since it is an
> esoteric use case where I would advise users to use a stateful DoFn instead,
> I think the bug fixed probably outweighs the bug introduced. I would like to
> fix for 2.1.0 but will report back soon.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
