[
https://issues.apache.org/jira/browse/BEAM-11087?focusedWorklogId=667322&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-667322
]
ASF GitHub Bot logged work on BEAM-11087:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 19/Oct/21 21:58
Start Date: 19/Oct/21 21:58
Worklog Time Spent: 10m
Work Description: lostluck commented on a change in pull request #15743:
URL: https://github.com/apache/beam/pull/15743#discussion_r732273445
##########
File path: sdks/go/pkg/beam/pardo.go
##########
@@ -55,7 +55,7 @@ func TryParDo(s Scope, dofn interface{}, col PCollection,
opts ...Option) ([]PCo
sideNode := s.Input.n
sideWfn := sideNode.WindowingStrategy().Fn
if sideWfn.Kind == window.Sessions {
- return nil, fmt.Errorf("error with side input %d in
DoFn %v: PCollections using merging WindowFns are not supported as side inputs.
Consider re-windowing the side input PCollection before use", sideNode, fn)
+ return nil, fmt.Errorf("error with side input %v in
DoFn %v: PCollections using merging WindowFns are not supported as side inputs.
Consider re-windowing the side input PCollection before use", sideNode, fn)
}
if (inWfn.Kind == window.GlobalWindows) && (sideWfn.Kind !=
window.GlobalWindows) {
return nil, fmt.Errorf("main input %v is global
windowed in DoFn %v but side input %v is not, cannot map windows correctly.
Consider re-windowing the side input PCOllection before use", col.n, fn,
sideNode)
Review comment:
So the sideNode says what Type and such is being used, but doesn't say
"this is the 2nd side input into the DoFn", which can be retrieve by changing
the for loop to give you the index ala: `for i, s := range side {`
The type information is useful, but most of the rest of the node printout
information isn't useful for users to find out which side input is the problem.
The types aren't very visible to users at construction time, but "this is the
3rd side input" would be. Hence the earlier phrasing of "side input %d"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 667322)
Time Spent: 2h 10m (was: 2h)
> [Go SDK] Validate Side Input behavior WRT windows
> --------------------------------------------------
>
> Key: BEAM-11087
> URL: https://issues.apache.org/jira/browse/BEAM-11087
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-go
> Reporter: Robert Burke
> Assignee: Jack McCluskey
> Priority: P3
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> DoFns with Side inputs implicitly observe windows, as Side Inputs are scoped
> to a current window, a powerful feature of beam.
> The ideal would be to you could try to unit test this in the exec package,
> creating a fake side input adapter (or use the real one), to more directly
> target the implementation.
> [https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/sideinput.go#L34]
> and exercising the ParDo code for Side Input handling directly.
> [https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/pardo.go#L38]
>
> Then it should be possible to bolster that with appropriate mock windows to
> ensure that side inputs are configured correctly.
> Altnernatively this behavior could be tested and validated with an
> integration test against real runners due to the complexity around Side
> Inputs making unit testing a challenge. (While side input code could be
> tested in that fashion, it's likely dramatically simpler to do the
> integration test.)
> Some light tests with Side Inputs already exist, but they're purely in the
> Global Window. Add tests for non-global windows to ensure that Side Inputs
> are scoped correctly.
> Integration test directory:
> [https://github.com/apache/beam/tree/master/sdks/go/test/integration/primitives]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
