[
https://issues.apache.org/jira/browse/BEAM-7709?focusedWorklogId=274373&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-274373
]
ASF GitHub Bot logged work on BEAM-7709:
----------------------------------------
Author: ASF GitHub Bot
Created on: 09/Jul/19 20:39
Start Date: 09/Jul/19 20:39
Worklog Time Spent: 10m
Work Description: youngoli commented on issue #9015: [BEAM-7709] Re-use
node for explicit flattens
URL: https://github.com/apache/beam/pull/9015#issuecomment-509798598
Just so I understand this change, when handling a link between a transform
with a Flatten URN and one of its inputs: Before it would just create a Flatten
instance and set it as the link, now it also uses the same Flatten for every
other input to the PTransform, so that when those links get put into this
function later it exits early at line 320?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 274373)
Time Spent: 1h 20m (was: 1h 10m)
> Flattening multiple outputs of a ParDoN fails
> ---------------------------------------------
>
> Key: BEAM-7709
> URL: https://issues.apache.org/jira/browse/BEAM-7709
> Project: Beam
> Issue Type: Bug
> Components: sdk-go
> Affects Versions: Not applicable
> Reporter: Robert Burke
> Assignee: Robert Burke
> Priority: Major
> Fix For: Not applicable
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> If a user does a beam.ParDoN for pardo > 2 and then passes one or more of
> the outputs to a flatten, then if the flatten occurs SDK side, it currently
> creates multiple flatten nodes, which then triggers the downstream pardo (the
> DoFn that consumes the Flatten's output) to be initialized multiple times for
> a single bundle.
> The fix is to pre-emptively populate the input links with the first created
> flatten, so subsequent tracings of the plan use the same flatten node the
> same way the Go direct runner does[1]. That would happen in the exec
> translate code.
> [[1]
> https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/runners/direct/direct.go#L299|https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/runners/direct/direct.go#L299]
> [[2]
> https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/translate.go#L493|https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/translate.go#L493]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
