[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313138#comment-16313138 ] ASF GitHub Bot commented on BEAM-2140: -- aljoscha opened a new pull request #4348: [BEAM-2140] Fix

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170472#comment-16170472 ] Eugene Kirpichov commented on BEAM-2140: Also note that this support is critical for portable

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170471#comment-16170471 ] Eugene Kirpichov commented on BEAM-2140: Thanks Aljoscha! It would be awesome to get this working

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170437#comment-16170437 ] Aljoscha Krettek commented on BEAM-2140: As I commented on the PR, I have a solution that is

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170435#comment-16170435 ] ASF GitHub Bot commented on BEAM-2140: -- Github user aljoscha closed the pull request at:

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126416#comment-16126416 ] Eugene Kirpichov commented on BEAM-2140: Sorry for the delayed response. The output watermark

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16116972#comment-16116972 ] Aljoscha Krettek commented on BEAM-2140: [~jkff] [~kenn] [~lzljs3620320] Returning to this after a

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16070150#comment-16070150 ] ASF GitHub Bot commented on BEAM-2140: -- GitHub user aljoscha opened a pull request:

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16069802#comment-16069802 ] Aljoscha Krettek commented on BEAM-2140: Yep, in the Flink Runner the processing path for

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068957#comment-16068957 ] Kenneth Knowles commented on BEAM-2140: --- Yea, as you say SDF can play by its own rules, so you can

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068782#comment-16068782 ] Eugene Kirpichov commented on BEAM-2140: Conceptually, watermarks are for PCollections - lower

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068740#comment-16068740 ] Eugene Kirpichov commented on BEAM-2140: So, to elaborate on what Kenn said. We dug a bit deeper

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068561#comment-16068561 ] Kenneth Knowles commented on BEAM-2140: --- Yea. I think if timers are set with an event time timestamp,

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16067984#comment-16067984 ] Aljoscha Krettek commented on BEAM-2140: Cool, so for the side channel to affect things we just

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16067130#comment-16067130 ] Kenneth Knowles commented on BEAM-2140: --- OK. We also came up with a new thought that affect timers in

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16067018#comment-16067018 ] Aljoscha Krettek commented on BEAM-2140: Ok, I also fixed the issue of dropped timers by exhausting

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16066759#comment-16066759 ] Aljoscha Krettek commented on BEAM-2140: I think to unblock this we have to move the SplittableDoFn

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065663#comment-16065663 ] Kenneth Knowles commented on BEAM-2140: --- Yes, your last comment is the reason. If the watermark of

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065661#comment-16065661 ] Eugene Kirpichov commented on BEAM-2140: Working backwards from that, in the "read Pubsub topic

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065627#comment-16065627 ] Eugene Kirpichov commented on BEAM-2140: ...Or is the problem that the watermark of the output

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065626#comment-16065626 ] Eugene Kirpichov commented on BEAM-2140: _we can't advance the watermark as though it was

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065478#comment-16065478 ] Kenneth Knowles commented on BEAM-2140: --- Ah, it is true that the input element is still pending. That

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065429#comment-16065429 ] Eugene Kirpichov commented on BEAM-2140: Okay, I see that I misunderstood what the watermark hold

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065398#comment-16065398 ] Aljoscha Krettek commented on BEAM-2140: I need to look into things a bit but I think your comment

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065235#comment-16065235 ] Kenneth Knowles commented on BEAM-2140: --- I considered for a long time what should happen with

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065211#comment-16065211 ] Kenneth Knowles commented on BEAM-2140: --- Taking a step back, I don't feel strongly that it needs to

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065171#comment-16065171 ] Kenneth Knowles commented on BEAM-2140: --- I agree that the proposal of registering timers with an

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064098#comment-16064098 ] Kenneth Knowles commented on BEAM-2140: --- [~lzljs3620320] you are correct - the decision about whether

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064090#comment-16064090 ] Jingsong Lee commented on BEAM-2140: On 1: If the decision whether the window expired is output

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063594#comment-16063594 ] Kenneth Knowles commented on BEAM-2140: --- There is an open need to be able to set a timer with an

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063581#comment-16063581 ] Kenneth Knowles commented on BEAM-2140: --- A watermark holds constrains the output watermark, not the

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063548#comment-16063548 ] Eugene Kirpichov commented on BEAM-2140: On 1: the timer should not be dropped because ProcessFn is

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063129#comment-16063129 ] Kenneth Knowles commented on BEAM-2140: --- Dropping the processing-time timer is the intended behavior

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062881#comment-16062881 ] Aljoscha Krettek commented on BEAM-2140: [~lzljs3620320]/[~jkff] I looked into this again and I

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005465#comment-16005465 ] Eugene Kirpichov commented on BEAM-2140: Aljoscha - SDF code does not inspect watermarks. Here's

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16003002#comment-16003002 ] Jingsong Lee commented on BEAM-2140: 1. {{SplittableParDo}} will clear its State, in

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16002370#comment-16002370 ] Aljoscha Krettek commented on BEAM-2140: I think this is mostly right, yes.  1. Why should it not

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16002039#comment-16002039 ] Jingsong Lee commented on BEAM-2140: First, {{SplittableParDo}} should not wrap {{StatefulDoFnRunner}}.

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997815#comment-15997815 ] Aljoscha Krettek commented on BEAM-2140: Yes, that seems ok. > Fix SplittableDoFn ValidatesRunner

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997590#comment-15997590 ] Davor Bonaci commented on BEAM-2140: [~aljoscha], this shouldn't be release-blocking, right?