[
https://issues.apache.org/jira/browse/BEAM-8804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heejong Lee updated BEAM-8804:
------------------------------
Description: Currently, Beam model doesn't have any information on the
order of input/output PCollections from PTransforms. Therefore, PCollectionList
needs to be converted to PCollectionTuple when it goes across the
cross-language boundaries (or even in the same language, whenever it is
converted between in-memory object and proto) and it's impossible to recreate
PCollectionList from proto with the original order. The possible workaround is
just to use PCollectionTuple with integer id (starting from 0 like indexes)
instead of PCollectionList. In that case, we should first well-define how we
generate proto from PCollectionList since each SDK uses a different convention.
(was: Currently, Beam model doesn't have any information on the order of
output PCollections from PTransforms. So, PCollectionList needs to be converted
to PCollectionTuple when it goes across the cross-language boundary (or even in
the same language, when it is converted between in-memory object and proto).)
> PCollectionList support in cross-language transforms
> ----------------------------------------------------
>
> Key: BEAM-8804
> URL: https://issues.apache.org/jira/browse/BEAM-8804
> Project: Beam
> Issue Type: Improvement
> Components: beam-model
> Reporter: Heejong Lee
> Assignee: Heejong Lee
> Priority: Major
>
> Currently, Beam model doesn't have any information on the order of
> input/output PCollections from PTransforms. Therefore, PCollectionList needs
> to be converted to PCollectionTuple when it goes across the cross-language
> boundaries (or even in the same language, whenever it is converted between
> in-memory object and proto) and it's impossible to recreate PCollectionList
> from proto with the original order. The possible workaround is just to use
> PCollectionTuple with integer id (starting from 0 like indexes) instead of
> PCollectionList. In that case, we should first well-define how we generate
> proto from PCollectionList since each SDK uses a different convention.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
