[
https://issues.apache.org/jira/browse/BEAM-13626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyang Hu updated BEAM-13626:
-------------------------------
Description:
For:
pcol := ...
tmp := beam.CrossLanguage(..., {"foo": pcol}, ...}
out := beam.CrossLanguage(..., {"bar": tmp}, ...}
In the result model pipeline, the second expanded transform's inputs may hold a
map from local name to the original PCollection/node ID, even if the first
expansion has changed the output IDs.
Looks like a fix is to move the output name remapping after
mergeExpandedWithPipeline:
https://github.com/apache/beam/blob/928da54e1a95219bfbfa420fbf9c8fb5514fd683/sdks/go/pkg/beam/core/runtime/graphx/translate.go#L160-L163
was:
For:
pcol := ...
tmp := beam.CrossLanguage(..., map[string]PCollection{"foo": pcol}, ...}
out := beam.CrossLanguage(..., map[string]PCollection{"bar": tmp}, ...}
In the result model pipeline, the second expanded transform's inputs may hold a
map from local name to the original PCollection/node ID, even if the first
expansion has changed the output IDs.
Looks like a fix is to move the output name remapping after
mergeExpandedWithPipeline:
https://github.com/apache/beam/blob/928da54e1a95219bfbfa420fbf9c8fb5514fd683/sdks/go/pkg/beam/core/runtime/graphx/translate.go#L160-L163
> Outdated Transform.Inputs values when chaining expanded transforms
> ------------------------------------------------------------------
>
> Key: BEAM-13626
> URL: https://issues.apache.org/jira/browse/BEAM-13626
> Project: Beam
> Issue Type: Bug
> Components: sdk-go
> Reporter: Tianyang Hu
> Assignee: Tianyang Hu
> Priority: P2
>
> For:
> pcol := ...
> tmp := beam.CrossLanguage(..., {"foo": pcol}, ...}
> out := beam.CrossLanguage(..., {"bar": tmp}, ...}
> In the result model pipeline, the second expanded transform's inputs may hold
> a map from local name to the original PCollection/node ID, even if the first
> expansion has changed the output IDs.
> Looks like a fix is to move the output name remapping after
> mergeExpandedWithPipeline:
> https://github.com/apache/beam/blob/928da54e1a95219bfbfa420fbf9c8fb5514fd683/sdks/go/pkg/beam/core/runtime/graphx/translate.go#L160-L163
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
