[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=364051=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364051 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 27/Dec/19 22:31 Start Date: 27/Dec/19 22:31 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 364051) Time Spent: 9h 40m (was: 9.5h) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 9h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=362768=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362768 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 23/Dec/19 22:20 Start Date: 23/Dec/19 22:20 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-568594913 @robertwb I've rebased and resolved the conflicts. Do you mind help to merge this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 362768) Time Spent: 9.5h (was: 9h 20m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 9.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=362620=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362620 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 23/Dec/19 18:27 Start Date: 23/Dec/19 18:27 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#discussion_r360967814 ## File path: sdks/python/apache_beam/transforms/trigger.py ## @@ -1180,6 +1181,11 @@ def merge(_, to_be_merged, merge_result): # pylint: disable=no-self-argument for window in merged_away: state.clear_state(window, self.WATERMARK_HOLD) +# drop expired windows +for window in list(windows_to_elements.keys()): + if input_watermark > window.end + self.allowed_lateness: +del windows_to_elements[window] Review comment: Yes. Once it's fallen behind, it can be dropped at any point. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 362620) Time Spent: 9h 20m (was: 9h 10m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 9h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=361736=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-361736 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 20/Dec/19 17:04 Start Date: 20/Dec/19 17:04 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-568002327 @robertwb gentle ping. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 361736) Time Spent: 9h 10m (was: 9h) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 9h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=360636=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360636 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 17/Dec/19 01:03 Start Date: 17/Dec/19 01:03 Worklog Time Spent: 10m Work Description: y1chi commented on pull request #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#discussion_r358547481 ## File path: sdks/python/apache_beam/transforms/core.py ## @@ -2293,6 +2308,7 @@ def __init__(self, self.windowfn = windowfn self.triggerfn = triggerfn self.accumulation_mode = accumulation_mode +self.allowed_lateness = Duration.of(allowed_lateness) self.timestamp_combiner = ( timestamp_combiner or TimestampCombiner.OUTPUT_AT_EOW) self._is_default = ( Review comment: Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 360636) Time Spent: 9h (was: 8h 50m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 9h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=360635=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360635 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 17/Dec/19 01:02 Start Date: 17/Dec/19 01:02 Worklog Time Spent: 10m Work Description: y1chi commented on pull request #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#discussion_r358547481 ## File path: sdks/python/apache_beam/transforms/core.py ## @@ -2293,6 +2308,7 @@ def __init__(self, self.windowfn = windowfn self.triggerfn = triggerfn self.accumulation_mode = accumulation_mode +self.allowed_lateness = Duration.of(allowed_lateness) self.timestamp_combiner = ( timestamp_combiner or TimestampCombiner.OUTPUT_AT_EOW) self._is_default = ( Review comment: Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 360635) Time Spent: 8h 50m (was: 8h 40m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 8h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=360633=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360633 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 17/Dec/19 01:01 Start Date: 17/Dec/19 01:01 Worklog Time Spent: 10m Work Description: y1chi commented on pull request #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#discussion_r358547142 ## File path: sdks/python/apache_beam/transforms/core.py ## @@ -2293,6 +2308,7 @@ def __init__(self, self.windowfn = windowfn self.triggerfn = triggerfn self.accumulation_mode = accumulation_mode +self.allowed_lateness = Duration.of(allowed_lateness) self.timestamp_combiner = ( timestamp_combiner or TimestampCombiner.OUTPUT_AT_EOW) self._is_default = ( Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 360633) Time Spent: 8.5h (was: 8h 20m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 8.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=360634=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360634 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 17/Dec/19 01:02 Start Date: 17/Dec/19 01:02 Worklog Time Spent: 10m Work Description: y1chi commented on pull request #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#discussion_r358547373 ## File path: sdks/python/apache_beam/testing/data/trigger_transcripts.yaml ## @@ -118,6 +139,7 @@ broken_on: - SwitchingDirectRunner window_fn: Sessions(10) trigger_fn: AfterWatermark(early=AfterCount(2), late=AfterCount(3)) +allowed_lateness: 85 Review comment: Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 360634) Time Spent: 8h 40m (was: 8.5h) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 8h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=360625=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360625 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 17/Dec/19 00:49 Start Date: 17/Dec/19 00:49 Worklog Time Spent: 10m Work Description: y1chi commented on pull request #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#discussion_r358541559 ## File path: sdks/python/apache_beam/transforms/trigger.py ## @@ -1180,6 +1181,11 @@ def merge(_, to_be_merged, merge_result): # pylint: disable=no-self-argument for window in merged_away: state.clear_state(window, self.WATERMARK_HOLD) +# drop expired windows +for window in list(windows_to_elements.keys()): + if input_watermark > window.end + self.allowed_lateness: +del windows_to_elements[window] Review comment: Just to double check: for a WindowValue that is expiring before merging but not expiring after merging, is the expected behavior to garbage collect it anyways? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 360625) Time Spent: 8h 20m (was: 8h 10m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 8h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=360591=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360591 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 16/Dec/19 23:40 Start Date: 16/Dec/19 23:40 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#discussion_r358522911 ## File path: sdks/python/apache_beam/transforms/core.py ## @@ -2293,6 +2308,7 @@ def __init__(self, self.windowfn = windowfn self.triggerfn = triggerfn self.accumulation_mode = accumulation_mode +self.allowed_lateness = Duration.of(allowed_lateness) self.timestamp_combiner = ( timestamp_combiner or TimestampCombiner.OUTPUT_AT_EOW) self._is_default = ( Review comment: and allowed lateness == 0? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 360591) Time Spent: 8h 10m (was: 8h) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 8h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=360590=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360590 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 16/Dec/19 23:40 Start Date: 16/Dec/19 23:40 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#discussion_r358521810 ## File path: sdks/python/apache_beam/testing/data/trigger_transcripts.yaml ## @@ -118,6 +139,7 @@ broken_on: - SwitchingDirectRunner window_fn: Sessions(10) trigger_fn: AfterWatermark(early=AfterCount(2), late=AfterCount(3)) +allowed_lateness: 85 Review comment: Nit: 85 seems an odd value. An even 100 would be better. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 360590) Time Spent: 8h (was: 7h 50m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 8h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=360589=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360589 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 16/Dec/19 23:40 Start Date: 16/Dec/19 23:40 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#discussion_r358523914 ## File path: sdks/python/apache_beam/transforms/trigger.py ## @@ -1180,6 +1181,11 @@ def merge(_, to_be_merged, merge_result): # pylint: disable=no-self-argument for window in merged_away: state.clear_state(window, self.WATERMARK_HOLD) +# drop expired windows +for window in list(windows_to_elements.keys()): + if input_watermark > window.end + self.allowed_lateness: +del windows_to_elements[window] Review comment: This test is not on the element timestamp, but window end. It seems it would be cleaner to simply not add it up at line 1151. (We need it up there to avoid merging on "unseen" data as well). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 360589) Time Spent: 7h 50m (was: 7h 40m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 7h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=360515=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360515 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 16/Dec/19 22:15 Start Date: 16/Dec/19 22:15 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-566172906 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 360515) Time Spent: 7h 40m (was: 7.5h) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 7h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=360514=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360514 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 16/Dec/19 22:14 Start Date: 16/Dec/19 22:14 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-566269180 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 360514) Time Spent: 7.5h (was: 7h 20m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 7.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=360473=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360473 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 16/Dec/19 20:06 Start Date: 16/Dec/19 20:06 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-566221417 @robertwb Hi Robert, would you mind take another look? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 360473) Time Spent: 7h 20m (was: 7h 10m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 7h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=360404=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360404 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 16/Dec/19 18:01 Start Date: 16/Dec/19 18:01 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-566172906 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 360404) Time Spent: 7h (was: 6h 50m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 7h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=360405=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360405 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 16/Dec/19 18:01 Start Date: 16/Dec/19 18:01 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-565542800 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 360405) Time Spent: 7h 10m (was: 7h) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 7h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=359548=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-359548 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 13/Dec/19 17:58 Start Date: 13/Dec/19 17:58 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-565542800 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 359548) Time Spent: 6h 40m (was: 6.5h) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 6h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=359549=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-359549 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 13/Dec/19 17:58 Start Date: 13/Dec/19 17:58 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-565253504 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 359549) Time Spent: 6h 50m (was: 6h 40m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 6h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=359051=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-359051 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 13/Dec/19 00:51 Start Date: 13/Dec/19 00:51 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-564257726 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 359051) Time Spent: 6.5h (was: 6h 20m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 6.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=359050=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-359050 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 13/Dec/19 00:51 Start Date: 13/Dec/19 00:51 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-565253504 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 359050) Time Spent: 6h 20m (was: 6h 10m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 6h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=357541=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-357541 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 11/Dec/19 00:53 Start Date: 11/Dec/19 00:53 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-564327946 @robertwb I've moved the window filtering into trigger_driver.process_element() and it should look cleaner now, PTAL This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 357541) Time Spent: 6h 10m (was: 6h) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 6h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=357406=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-357406 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 10/Dec/19 21:05 Start Date: 10/Dec/19 21:05 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-564257726 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 357406) Time Spent: 6h (was: 5h 50m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 6h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=357327=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-357327 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 10/Dec/19 18:40 Start Date: 10/Dec/19 18:40 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-563464985 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 357327) Time Spent: 5h 40m (was: 5.5h) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 5h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=357326=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-357326 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 10/Dec/19 18:40 Start Date: 10/Dec/19 18:40 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-564173571 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 357326) Time Spent: 5.5h (was: 5h 20m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 5.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=357328=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-357328 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 10/Dec/19 18:40 Start Date: 10/Dec/19 18:40 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-564173571 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 357328) Time Spent: 5h 50m (was: 5h 40m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 5h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=356612=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356612 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 09/Dec/19 22:15 Start Date: 09/Dec/19 22:15 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-563464985 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356612) Time Spent: 5h 10m (was: 5h) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 5h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=356613=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356613 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 09/Dec/19 22:15 Start Date: 09/Dec/19 22:15 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-562765769 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356613) Time Spent: 5h 20m (was: 5h 10m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 5h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=355484=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-355484 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 06/Dec/19 22:40 Start Date: 06/Dec/19 22:40 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-562765769 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 355484) Time Spent: 5h (was: 4h 50m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=354762=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354762 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 05/Dec/19 23:06 Start Date: 05/Dec/19 23:06 Worklog Time Spent: 10m Work Description: robertwb commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-562358478 I can buy the argument that we could be more consistent with Java (though that does mean that we'll be duplicating more code everywhere). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 354762) Time Spent: 4h 50m (was: 4h 40m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 4h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[
https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=354761=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354761
]
ASF GitHub Bot logged work on BEAM-8824:
Author: ASF GitHub Bot
Created on: 05/Dec/19 23:05
Start Date: 05/Dec/19 23:05
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10216: [BEAM-8824]
Add support to allow specify window allowed_lateness in python sdk
URL: https://github.com/apache/beam/pull/10216#discussion_r354593245
##
File path: sdks/python/apache_beam/testing/data/trigger_transcripts.yaml
##
@@ -41,6 +42,27 @@ transcript:
- expect:
- {window: [0, 9], values: [1, 2, 3, 7], timestamp: 9, late: true}
+---
+name: fixed_drop_late_data_after_allowed_lateness
+window_fn: FixedWindows(10)
+trigger_fn: AfterWatermark(early=AfterCount(3), late=AfterCount(1))
+skip_trigger_transcript_test: True # trigger process_element doesn't take
allowed_lateness into account
Review comment:
If we're going to use this, re-use the broken-on tag.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 354761)
Time Spent: 4h 40m (was: 4.5h)
> Add support for allowed lateness in python sdk
> --
>
> Key: BEAM-8824
> URL: https://issues.apache.org/jira/browse/BEAM-8824
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Yichi Zhang
>Assignee: Yichi Zhang
>Priority: Major
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=354665=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354665 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 05/Dec/19 20:46 Start Date: 05/Dec/19 20:46 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-562310477 Gentle ping for another look. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 354665) Time Spent: 4.5h (was: 4h 20m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 4.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=353157=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-353157 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 04/Dec/19 01:01 Start Date: 04/Dec/19 01:01 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-561427163 > Yes, we should just add it to `driver.process_elements`. doing it in StreamingGroupAlsoByWindowEvaluator seems to be more consistent with Java behavior https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java#L206 should we just skip the trigger test with allowed lateness? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 353157) Time Spent: 4h 20m (was: 4h 10m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 4h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=353149=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-353149 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 04/Dec/19 00:51 Start Date: 04/Dec/19 00:51 Worklog Time Spent: 10m Work Description: y1chi commented on pull request #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#discussion_r353500374 ## File path: sdks/python/apache_beam/transforms/core.py ## @@ -2265,10 +2266,18 @@ def __init__(self, windowfn, triggerfn=None, accumulation_mode=None, raise ValueError( 'window fn (%s) does not have a determanistic coder (%s)' % ( windowfn, windowfn.get_window_coder())) +if isinstance(allowed_lateness, Duration): Review comment: got it, thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 353149) Time Spent: 4h 10m (was: 4h) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 4h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=353123=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-353123 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 04/Dec/19 00:04 Start Date: 04/Dec/19 00:04 Worklog Time Spent: 10m Work Description: robertwb commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-561413476 Yes, we should just add it to `driver.process_elements`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 353123) Time Spent: 3h 50m (was: 3h 40m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 3h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=353124=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-353124 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 04/Dec/19 00:04 Start Date: 04/Dec/19 00:04 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#discussion_r353487442 ## File path: sdks/python/apache_beam/transforms/core.py ## @@ -2265,10 +2266,18 @@ def __init__(self, windowfn, triggerfn=None, accumulation_mode=None, raise ValueError( 'window fn (%s) does not have a determanistic coder (%s)' % ( windowfn, windowfn.get_window_coder())) +if isinstance(allowed_lateness, Duration): Review comment: I think we can just do `Duration.of(allowed_lateness)`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 353124) Time Spent: 4h (was: 3h 50m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 4h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=353006=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-353006 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 03/Dec/19 21:39 Start Date: 03/Dec/19 21:39 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-561369002 @robertwb I've addressed the comments, PTAL, thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 353006) Time Spent: 3h 40m (was: 3.5h) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 3h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=352841=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352841 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 03/Dec/19 18:22 Start Date: 03/Dec/19 18:22 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-561293895 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352841) Time Spent: 3.5h (was: 3h 20m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=352282=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352282 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 02/Dec/19 22:59 Start Date: 02/Dec/19 22:59 Worklog Time Spent: 10m Work Description: y1chi commented on pull request #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#discussion_r352904529 ## File path: sdks/python/apache_beam/transforms/trigger_test.py ## @@ -718,6 +719,9 @@ def fire_timers(): output = [ _windowed_value_info(wv) for wv in driver.process_elements(state, bundle, watermark)] +# ignore WindowValues with expired window in output +output = [wvi for wvi in output + if wvi['window'][1] + allowed_lateness >= watermark] Review comment: or we can probably use driver.process_elements to deal with the lateness if we can add input watermark to the function arg list, I'll see if that's possible, it is potentially a larger change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352282) Time Spent: 3h 20m (was: 3h 10m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=352280=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352280 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 02/Dec/19 22:57 Start Date: 02/Dec/19 22:57 Worklog Time Spent: 10m Work Description: y1chi commented on pull request #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#discussion_r352904529 ## File path: sdks/python/apache_beam/transforms/trigger_test.py ## @@ -718,6 +719,9 @@ def fire_timers(): output = [ _windowed_value_info(wv) for wv in driver.process_elements(state, bundle, watermark)] +# ignore WindowValues with expired window in output +output = [wvi for wvi in output + if wvi['window'][1] + allowed_lateness >= watermark] Review comment: or we can probably use driver.process_elements to deal with the lateness if we can add input watermark to the function arg list, I'll see if that's possible. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352280) Time Spent: 3h 10m (was: 3h) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=352262=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352262 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 02/Dec/19 22:11 Start Date: 02/Dec/19 22:11 Worklog Time Spent: 10m Work Description: y1chi commented on pull request #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#discussion_r352886273 ## File path: sdks/python/apache_beam/transforms/core.py ## @@ -2235,7 +2235,20 @@ def expand(self, pcoll): class Windowing(object): def __init__(self, windowfn, triggerfn=None, accumulation_mode=None, - timestamp_combiner=None): + timestamp_combiner=None, allowed_lateness=0): +"""Class representing the window strategy. + +Args: + windowfn: Window assign function. + triggerfn: Trigger function. + accumulation_mode: a AccumulationMode, controls what to do with data +when a trigger fires multiple times. + timestamp_combiner: a TimestampCombiner, determines how output +timestamps of grouping operations are assigned. + allowed_lateness(int): Maximum delay in seconds after end of window Review comment: yeah, thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352262) Time Spent: 3h (was: 2h 50m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=352261=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352261 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 02/Dec/19 22:11 Start Date: 02/Dec/19 22:11 Worklog Time Spent: 10m Work Description: y1chi commented on pull request #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#discussion_r352886109 ## File path: sdks/python/apache_beam/transforms/trigger_test.py ## @@ -718,6 +719,9 @@ def fire_timers(): output = [ _windowed_value_info(wv) for wv in driver.process_elements(state, bundle, watermark)] +# ignore WindowValues with expired window in output +output = [wvi for wvi in output + if wvi['window'][1] + allowed_lateness >= watermark] Review comment: Seems that the java direct runner and dataflow JRH both filters elements before trigger so I thought it should be better to do it the same way in python. Filter out the elements here is more for reusing of the transcripts in trigger tests since here trigger doesn't really deal with lateness. Another option is to pass the transcript tests that specifies allowed_lateness in trigger tests, would that makes more sense? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352261) Time Spent: 2h 50m (was: 2h 40m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=352247=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352247 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 02/Dec/19 21:52 Start Date: 02/Dec/19 21:52 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#discussion_r352875296 ## File path: sdks/python/apache_beam/transforms/core.py ## @@ -2315,7 +2331,8 @@ def from_runner_api(proto, context): windowfn=WindowFn.from_runner_api(proto.window_fn, context), triggerfn=TriggerFn.from_runner_api(proto.trigger, context), accumulation_mode=proto.accumulation_mode, -timestamp_combiner=proto.output_time) +timestamp_combiner=proto.output_time, +allowed_lateness=proto.allowed_lateness // 1000) Review comment: This is lossy, and it's possible that this comes from other languages that support sub-second allowed lateness. `Duration(micros=proto.allowed_lateness * 1000)`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352247) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=352244=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352244 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 02/Dec/19 21:52 Start Date: 02/Dec/19 21:52 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#discussion_r352870955 ## File path: sdks/python/apache_beam/runners/direct/transform_evaluator.py ## @@ -816,10 +816,15 @@ def process_element(self, element): timer_firing.window, timer_firing.name, timer_firing.time_domain, timer_firing.timestamp, state): self.gabw_items.append(wvalue.with_value((k, wvalue.value))) -watermark = self._evaluation_context._watermark_manager.get_watermarks( -self._applied_ptransform).output_watermark +watermark = (self._evaluation_context._watermark_manager.get_watermarks( Review comment: Don't need added ()'s. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352244) Time Spent: 2.5h (was: 2h 20m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=352245=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352245 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 02/Dec/19 21:52 Start Date: 02/Dec/19 21:52 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#discussion_r352874522 ## File path: sdks/python/apache_beam/transforms/core.py ## @@ -2235,7 +2235,20 @@ def expand(self, pcoll): class Windowing(object): def __init__(self, windowfn, triggerfn=None, accumulation_mode=None, - timestamp_combiner=None): + timestamp_combiner=None, allowed_lateness=0): +"""Class representing the window strategy. + +Args: + windowfn: Window assign function. + triggerfn: Trigger function. + accumulation_mode: a AccumulationMode, controls what to do with data +when a trigger fires multiple times. + timestamp_combiner: a TimestampCombiner, determines how output +timestamps of grouping operations are assigned. + allowed_lateness(int): Maximum delay in seconds after end of window Review comment: Nit: I'd cut the (int) for consistency. Technically we should accept floats here as well, and possibly convert to Durations. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352245) Time Spent: 2h 40m (was: 2.5h) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=352246=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352246 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 02/Dec/19 21:52 Start Date: 02/Dec/19 21:52 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#discussion_r352877739 ## File path: sdks/python/apache_beam/transforms/trigger_test.py ## @@ -718,6 +719,9 @@ def fire_timers(): output = [ _windowed_value_info(wv) for wv in driver.process_elements(state, bundle, watermark)] +# ignore WindowValues with expired window in output +output = [wvi for wvi in output + if wvi['window'][1] + allowed_lateness >= watermark] Review comment: Above we do filtering before calling driver.process_elements. Here we do filtering after, which seems inconsistent. Any reason allowed_lateness shouldn't be used within `driver.process_elements` (which takes windowing as a parameter) to consistently do the right thing? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352246) Time Spent: 2h 40m (was: 2.5h) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=352243=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352243 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 02/Dec/19 21:52 Start Date: 02/Dec/19 21:52 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#discussion_r352873760 ## File path: sdks/python/apache_beam/runners/direct/transform_evaluator.py ## @@ -834,6 +839,15 @@ def finish_bundle(self): return TransformResult(self, bundles, [], None, self.keyed_holds) + def _drop_expired_windows(self, wvalue, input_watermark, windowing): +allowed_lateness = windowing.allowed_lateness + +def is_not_expired(window): + return input_watermark - allowed_lateness <= window.end + +wvalue.windows = [w for w in wvalue.windows if is_not_expired(w)] Review comment: Please don't modify the input; return a new value instead. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352243) Time Spent: 2h 20m (was: 2h 10m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=350694=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350694 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 27/Nov/19 21:50 Start Date: 27/Nov/19 21:50 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-559262483 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350694) Time Spent: 2h 10m (was: 2h) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=350688=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350688 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 27/Nov/19 21:39 Start Date: 27/Nov/19 21:39 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-559216682 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350688) Time Spent: 1h 50m (was: 1h 40m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=350689=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350689 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 27/Nov/19 21:39 Start Date: 27/Nov/19 21:39 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-559259855 R: @robertwb @aaltay This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350689) Time Spent: 2h (was: 1h 50m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=350637=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350637 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 27/Nov/19 19:11 Start Date: 27/Nov/19 19:11 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-559216682 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350637) Time Spent: 1h 40m (was: 1.5h) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=350155=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350155 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 27/Nov/19 01:32 Start Date: 27/Nov/19 01:32 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-558878136 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350155) Time Spent: 1h 20m (was: 1h 10m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=350156=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350156 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 27/Nov/19 01:32 Start Date: 27/Nov/19 01:32 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-558860803 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350156) Time Spent: 1.5h (was: 1h 20m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=350122=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350122 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 27/Nov/19 00:43 Start Date: 27/Nov/19 00:43 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-558878136 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350122) Time Spent: 1h 10m (was: 1h) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=350105=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350105 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 26/Nov/19 23:26 Start Date: 26/Nov/19 23:26 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-558860803 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350105) Time Spent: 1h (was: 50m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=350102=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350102 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 26/Nov/19 23:13 Start Date: 26/Nov/19 23:13 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-558819220 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350102) Time Spent: 50m (was: 40m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=350026=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350026 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 26/Nov/19 21:12 Start Date: 26/Nov/19 21:12 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-558819220 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350026) Time Spent: 40m (was: 0.5h) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=350025=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350025 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 26/Nov/19 21:12 Start Date: 26/Nov/19 21:12 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-558819057 retest this This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350025) Time Spent: 0.5h (was: 20m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=350024=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350024 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 26/Nov/19 21:12 Start Date: 26/Nov/19 21:12 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-558819057 retest this This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350024) Time Spent: 20m (was: 10m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=349512=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-349512 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 26/Nov/19 02:23 Start Date: 26/Nov/19 02:23 Worklog Time Spent: 10m Work Description: y1chi commented on pull request #10216: [BEAM-8824] Add support to specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[