[ https://issues.apache.org/jira/browse/BEAM-4826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566235#comment-16566235 ]
Henning Rohde commented on BEAM-4826: ------------------------------------- Dataflow drops the flatten altogether. > Flink runner sends bad flatten to SDK > ------------------------------------- > > Key: BEAM-4826 > URL: https://issues.apache.org/jira/browse/BEAM-4826 > Project: Beam > Issue Type: Bug > Components: runner-flink > Reporter: Henning Rohde > Assignee: Ankur Goenka > Priority: Major > Labels: portability > > For a Go flatten test w/ 3 input, the Flink runner splits this into 3 bundle > descriptors. But it sends the original 3-input flatten but w/ 1 actual input > present in each bundle descriptor. This is inconsistent and the SDK shouldn't > expect dangling PCollections. In contrast, Dataflow removes the flatten when > it does the same split. > Snippet: > register: < > process_bundle_descriptor: < > id: "3" > transforms: < > key: "e4" > value: < > unique_name: "github.com/apache/beam/sdks/go/pkg/beam.createFn'1" > spec: < > urn: "urn:beam:transform:pardo:v1" > payload: [...] > > > inputs: < > key: "i0" > value: "n3" > > > outputs: < > key: "i0" > value: "n4" > > > > > > > transforms: < > key: "e7" > value: < > unique_name: "Flatten" > spec: < > urn: "beam:transform:flatten:v1" > > > inputs: < > key: "i0" > value: "n2" > > > inputs: < > key: "i1" > value: "n4" . // <----------- only one present. > > > inputs: < > key: "i2" > value: "n6" > > > outputs: < > key: "i0" > value: "n7" > > > > > > > [...] -- This message was sent by Atlassian JIRA (v7.6.3#76005)