[
https://issues.apache.org/jira/browse/BEAM-7709?focusedWorklogId=274359&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-274359
]
ASF GitHub Bot logged work on BEAM-7709:
----------------------------------------
Author: ASF GitHub Bot
Created on: 09/Jul/19 20:30
Start Date: 09/Jul/19 20:30
Worklog Time Spent: 10m
Work Description: lukecwik commented on issue #9015: [BEAM-7709] Re-use
node for explicit flattens
URL: https://github.com/apache/beam/pull/9015#issuecomment-509795693
It is valid to have multiple copies of downstream DoFns but it adds a lot of
complexity to do it right and is unlikely to be performant. (The complexity
comes from things like user state/timers expectations that readers/writers have
a consistent view per key).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 274359)
Time Spent: 1h (was: 50m)
> Flattening multiple outputs of a ParDoN fails
> ---------------------------------------------
>
> Key: BEAM-7709
> URL: https://issues.apache.org/jira/browse/BEAM-7709
> Project: Beam
> Issue Type: Bug
> Components: sdk-go
> Affects Versions: Not applicable
> Reporter: Robert Burke
> Assignee: Robert Burke
> Priority: Major
> Fix For: Not applicable
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> If a user does a beam.ParDoN for pardo > 2 and then passes one or more of
> the outputs to a flatten, then if the flatten occurs SDK side, it currently
> creates multiple flatten nodes, which then triggers the downstream pardo (the
> DoFn that consumes the Flatten's output) to be initialized multiple times for
> a single bundle.
> The fix is to pre-emptively populate the input links with the first created
> flatten, so subsequent tracings of the plan use the same flatten node the
> same way the Go direct runner does[1]. That would happen in the exec
> translate code.
> [[1]
> https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/runners/direct/direct.go#L299|https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/runners/direct/direct.go#L299]
> [[2]
> https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/translate.go#L493|https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/translate.go#L493]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
