[
https://issues.apache.org/jira/browse/BEAM-9382?focusedWorklogId=399919&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-399919
]
ASF GitHub Bot logged work on BEAM-9382:
----------------------------------------
Author: ASF GitHub Bot
Created on: 08/Mar/20 22:59
Start Date: 08/Mar/20 22:59
Worklog Time Spent: 10m
Work Description: acrites commented on issue #10988: [BEAM-9382] Clean up
of TestStreamTranscriptTests.
URL: https://github.com/apache/beam/pull/10988#issuecomment-596263133
RE fixing the Python direct runner: right now we just use the return value
from on_fire for each trigger to set the final bit. This doesn't take into
account allowed lateness. So we'd probably need to update all of the
triggers.
Maybe just a single test for now that is disabled and has the final bit
check? Or did you want to duplicate all the tests? I'm thinking something
like this one:
name: fixed_allowed_lateness_final_set
broken_on:
- SwitchingDirectRunner # Doesn't set final field properly with late
data.
window_fn: FixedWindows(10)
trigger_fn: AfterWatermark(early=AfterCount(2), late=AfterCount(1))
timestamp_combiner: OUTPUT_AT_EOW
allowed_lateness: 10
transcript:
- input: [1, 2]
- expect:
- {window: [0, 9], values: [1, 2], timestamp: 9, final: false}
- input: [3]
- watermark: 100
- expect:
- {window: [0, 9], values: [1, 2, 3], timestamp: 9, final: true}
On Fri, Mar 6, 2020 at 11:45 AM Robert Bradshaw <[email protected]>
wrote:
> The Python direct runner does support PaneInfo, but it doesn't (yet)
> support allowed lateness, so the "final" bit is wrong. Yes, we should fix
> this, though hopefully on the FnApiRunner once it supports streaming rather
> than invest too much into the old direct runner (unless it's cheap to do).
>
> I would rather disable the "final" bit check for these tests (and add a
> new, disabled test that checks this bit) than disable these tests entirely
> for the direct runner. So as is, this change LGTM.
>
> We could also consider expanding the API to indicate certain features are
> broken on certain runners, rather than entire tests, which would allow us
> to fully specify the expected results and still produce partial validation
> on incomplete/in-progress runners.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
>
<https://github.com/apache/beam/pull/10988?email_source=notifications&email_token=ACOKCJXEORU3BA5EY2IWAFTRGFHD7A5CNFSM4K5AFZM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOCS4KY#issuecomment-595930667>,
> or unsubscribe
>
<https://github.com/notifications/unsubscribe-auth/ACOKCJR3YOU4QZIDUJOT7DLRGFHD7ANCNFSM4K5AFZMQ>
> .
>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 399919)
Time Spent: 1h 20m (was: 1h 10m)
> TestStreamTranscriptTest relies on non-deterministic behavior
> -------------------------------------------------------------
>
> Key: BEAM-9382
> URL: https://issues.apache.org/jira/browse/BEAM-9382
> Project: Beam
> Issue Type: Bug
> Components: testing
> Reporter: Andrew Crites
> Assignee: Andrew Crites
> Priority: Minor
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The test discarding_early_fixed uses an early trigger Count(2) and then
> inserts 3 elements, assuming all 3 will get emitted in the early pane.
> However, runners do not have to follow this behavior. Instead, they could
> emit the first two elements seen and then buffer the third until something
> else comes in. We should change this test to only insert 2 elements so that
> all runners will behave the same.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
