[
https://issues.apache.org/jira/browse/BEAM-11084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Burke updated BEAM-11084:
--------------------------------
Description:
While Windowing basics are implemented in the Go SDK, there is precious little
validation that it's correct. Much work has been done largely for the benefit
of Global Windowing.
DoFn Authors can specify event times for elements by returning a
`beam.EventTime` along with their element, allowing windowing transforms by the
runner to take effect. They can observe windows in their DoFns by having a
`[]beam.Window` parameter, and extract results.
Pipeline authors can insert `beam.WindowInto` transforms to have elements be
batched into the appropriate window slices.
But very little of this has been validated end to end. This umbrella task is
to test and validate specific aspects of the window implementation to ensure
correctness.
The tricky part is that windows are selected by runners, meaning many of these
must be integration tests.
* Windows are correctly observed in DoFns.
* Validate windows are aggregated correctly in GBKs and Lifted Combines
** In particular sliding windows where a single element could contribute to
multiple aggregates
* Validate Side Input behavior WRT windows (implicit observation of windows)
** In particular, Side Inputs are valid per window.
was:
While Windowing basics are implemented in the Go SDK, there is precious little
validation that it's correct. Much work has been done largely for the benefit
of Global Windowing.
DoFn Authors can specify event times for elements by returning a
`beam.EventTime` along with their element, allowing windowing transforms by the
runner to take effect. They can observe windows in their DoFns by having a
`[]beam.Window` parameter, and extract results.
Pipeline authors can insert `beam.WindowInto` transforms to have elements be
batched into the appropriate window slices.
But very little of this has been validated end to end. This umbrella task is
to test and validate specific aspects of the window implementation to ensure
correctness.
The tricky part is that windows are selected by runners, meaning many of these
must be integration tests.
* Windows are correctly observed in DoFns.
* Validate windows are aggregated correctly in GBKs
* Validate that multi-window lifted combiners
** In particular sliding windows where a single element could contribute to
multiple aggregates
* Validate Side Input behavior WRT windows (implicit observation of windows)
** In particular, Side Inputs are valid per window.
> [Go SDK] Validate Windowing
> ---------------------------
>
> Key: BEAM-11084
> URL: https://issues.apache.org/jira/browse/BEAM-11084
> Project: Beam
> Issue Type: Task
> Components: sdk-go
> Reporter: Robert Burke
> Priority: P3
>
> While Windowing basics are implemented in the Go SDK, there is precious
> little validation that it's correct. Much work has been done largely for the
> benefit of Global Windowing.
> DoFn Authors can specify event times for elements by returning a
> `beam.EventTime` along with their element, allowing windowing transforms by
> the runner to take effect. They can observe windows in their DoFns by having
> a `[]beam.Window` parameter, and extract results.
> Pipeline authors can insert `beam.WindowInto` transforms to have elements be
> batched into the appropriate window slices.
> But very little of this has been validated end to end. This umbrella task is
> to test and validate specific aspects of the window implementation to ensure
> correctness.
> The tricky part is that windows are selected by runners, meaning many of
> these must be integration tests.
> * Windows are correctly observed in DoFns.
> * Validate windows are aggregated correctly in GBKs and Lifted Combines
> ** In particular sliding windows where a single element could contribute to
> multiple aggregates
> * Validate Side Input behavior WRT windows (implicit observation of windows)
> ** In particular, Side Inputs are valid per window.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
