[
https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116972#comment-16116972
]
Aljoscha Krettek commented on BEAM-2140:
----------------------------------------
[~jkff] [~kenn] [~lzljs3620320] Returning to this after a bit of a break. I
did a simpler implementation that does not duplicate the {{ProcessFn}} code but
instead blocks shutdown in the {{DoFnOperator}} while there are pending
processing-time timers.
This solution works but is still flaky in tests (and in edge cases in the real
world). The reason is still that processing-time timers don't hold back any
watermark (neither the input nor the output watermark, if I'm correct). The
situation is this (in the basic tests): we have a {{Create}}, a {{SDF}}, and a
validating {{PAssert}}
{{code}}
Create -> SDF -> PAssert
{{code}}
In failure cases this happens: 1) {{Create}} emits some elements, 2) the
{{SDF}} processes some elements, then yields and a processing-time timer is set
for processing the remainder of the restriction, 3) {{Create}} finishes, the
watermark goes to +Inf 4) the watermark "passes" (un-held) thought the SDF, 4)
the watermark triggers computation at the {{PAssert}} and this notices that we
didn't receive all expected data.
How does this work in the Dataflow runner if processing-time timers don't hold
back the watermark. Or is there a custom implementation for SDF in the Dataflow
runner?
> Fix SplittableDoFn ValidatesRunner tests in FlinkRunner
> -------------------------------------------------------
>
> Key: BEAM-2140
> URL: https://issues.apache.org/jira/browse/BEAM-2140
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
> Reporter: Aljoscha Krettek
> Assignee: Aljoscha Krettek
>
> As discovered as part of BEAM-1763, there is a failing SDF test. We disabled
> the tests to unblock the open PR for BEAM-1763.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
