[
https://issues.apache.org/jira/browse/BEAM-8581?focusedWorklogId=342314&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-342314
]
ASF GitHub Bot logged work on BEAM-8581:
----------------------------------------
Author: ASF GitHub Bot
Created on: 13/Nov/19 00:41
Start Date: 13/Nov/19 00:41
Worklog Time Spent: 10m
Work Description: rohdesamuel commented on pull request #10035:
[BEAM-8581] and [BEAM-8582] watermark and trigger fixes
URL: https://github.com/apache/beam/pull/10035#discussion_r345519367
##########
File path: sdks/python/apache_beam/testing/test_stream_test.py
##########
@@ -195,15 +195,12 @@ def test_gbk_execution_after_watermark_trigger(self):
# assert per window
expected_window_to_elements = {
- window.IntervalWindow(15, 30): [
- ('k', ['a']),
- ('k', []),
- ],
+ window.IntervalWindow(0, 15): [('k', ['a']), ('k', [])],
}
assert_that(
records,
equal_to_per_window(expected_window_to_elements),
- use_global_window=False,
+ reify_windows=True,
Review comment:
Basically, I want to define the equal_to_per_window to test on the emitted
windows all at once, at the end of the test. This makes the matcher less
stateful in its implementation. The bug was originally found in the
test_stream_test. In fact, the only one that properly catches the bug that this
PR fixes is the TestStreamTest.test_gbk_execution_after_watermark_trigger.
Hence, why I added test_stream_test and util fixes to this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 342314)
Time Spent: 3h 50m (was: 3h 40m)
> Python SDK labels ontime empty panes as late
> --------------------------------------------
>
> Key: BEAM-8581
> URL: https://issues.apache.org/jira/browse/BEAM-8581
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Reporter: Sam Rohde
> Assignee: Sam Rohde
> Priority: Major
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> The GeneralTriggerDriver does not put watermark holds on timers, leading to
> the ontime empty pane being considered late data.
> Fix: Add a new notion of whether a trigger has an ontime pane. If it does,
> then set a watermark hold to end of window - 1.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
