[
https://issues.apache.org/jira/browse/BEAM-2571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16080623#comment-16080623
]
Aljoscha Krettek commented on BEAM-2571:
----------------------------------------
[~kenn] with some super ugly debugging I think I found at least one problem:
the Flink and Beam watermarks have slightly different semantics. In Beam, a
watermark {{t}} says there won't be elements with a timestamp {{< t}} in the
future. In Flink a watermark {{t}} says there won't be elements with a
timestamp {{<= t}} in the future. This means, that Flink will fire a timer for
time {{t0}} when the current input watermark is at least {{t0}}. The default
Trigger, however, has this piece of code:
https://github.com/apache/beam/blob/ca41af8fe4711ab4a81c2a33746a64e64fb0ca37/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/DefaultTriggerStateMachine.java#L76-L76.
Meaning it will only fire when the input watermark is past {{t0}}. What this
means here, is that we fire the only timer that was set but the Trigger doesn't
think it's time yet and so doesn't fire.
I'm not yet sure how to solve this and if it is the only problem. A solution
might be to shift everything that is timestamp related in the Flink runner by
one, I'd have to first find all the places, though, and put in very good tests.
This is the branch that I used for debugging:
https://github.com/aljoscha/beam/tree/jira-2571-sleuthing-for-combinetest
> Flink ValidatesRunner failing CombineTest.testSlidingWindowsCombineWithContext
> ------------------------------------------------------------------------------
>
> Key: BEAM-2571
> URL: https://issues.apache.org/jira/browse/BEAM-2571
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
> Reporter: Kenneth Knowles
> Assignee: Aljoscha Krettek
> Fix For: 2.1.0
>
>
> This appears to have been caused by https://github.com/apache/beam/pull/3429
> which fixes a couple errors in how trigger timers were processed / final
> panes labeled.
> I am investigating, considering roll back vs forward fix. Since it is an
> esoteric use case where I would advise users to use a stateful DoFn instead,
> I think the bug fixed probably outweighs the bug introduced. I would like to
> fix for 2.1.0 but will report back soon.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
