[
https://issues.apache.org/jira/browse/BEAM-11715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17275449#comment-17275449
]
Yifan Mai commented on BEAM-11715:
----------------------------------
Dataflow with portable job submission _should_ work because
[https://github.com/apache/beam/pull/13829] uses a hacky workaround. We convert
the pipeline to proto, do the optimizations, convert it back into pipeline, use
CombinePerKeyPTransformOverride to perform the expansion, and then convert back
into proto. Note that we convert the pipeline to the proto _twice_.
I can try running the 2.28 branch with --use_runner_v2 (is that all that is
needed to trigger portable job submission?) and report back.
> Combiner packing creates an incorrect proto
> -------------------------------------------
>
> Key: BEAM-11715
> URL: https://issues.apache.org/jira/browse/BEAM-11715
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Reporter: Chamikara Madhusanka Jayalath
> Assignee: Yifan Mai
> Priority: P0
> Fix For: 2.28.0
>
>
> Seems like optimization creates a CombinePerKey transform that does not have
> any sub-transforms. We should fix this by preserving the old structure for
> CombinePerKey (GBK+CombineValues composite).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
