[ https://issues.apache.org/jira/browse/BEAM-4582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Luke Cwik resolved BEAM-4582. ----------------------------- Fix Version/s: 2.21.0 Assignee: Luke Cwik Resolution: Cannot Reproduce Streaming create for portable Dataflow now works. > Incorrectly translates > apache_beam.runners.dataflow.native_io.streaming_create.DecodeAndEmitDoFn > when creating the Dataflow pipeline json description > ----------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: BEAM-4582 > URL: https://issues.apache.org/jira/browse/BEAM-4582 > Project: Beam > Issue Type: Bug > Components: runner-dataflow > Reporter: Luke Cwik > Assignee: Luke Cwik > Priority: P2 > Labels: portability, stale-P2 > Fix For: 2.21.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > When executing against Dataflow, the JSON pipeline description contains the > following JSON which doesn't appear in the pipeline proto: > > {code:java} > { > "kind": "ParallelDo", > "name": "s2", > "properties": { > "display_data": [ > { > "key": "fn", > "label": "Transform Function", > "namespace": "apache_beam.transforms.core.ParDo", > "shortValue": "DecodeAndEmitDoFn", > "type": "STRING", > "value": > "apache_beam.runners.dataflow.native_io.streaming_create.DecodeAndEmitDoFn" > } > ], > "non_parallel_inputs": {}, > "output_info": [ > { > "encoding": { > "@type": "kind:windowed_value", > "component_encodings": [ > { > "@type": > "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/", > > "component_encodings": [ > { > "@type": > "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/", > > "component_encodings": [] > }, > { > "@type": > "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/", > > "component_encodings": [] > } > ], > "is_pair_like": true > }, > { > "@type": "kind:global_window" > } > ], > "is_wrapper": true > }, > "output_name": "out", > "user_name": "Some Numbers/Decode Values.out" > } > ], > "parallel_input": { > "@type": "OutputReference", > "output_name": "out", > "step_name": "s1" > }, > "serialized_fn": "ref_AppliedPTransform_AppliedPTransform_45", > "user_name": "Some Numbers/Decode Values" > } > }, > {code} > This causes the DataflowRunner to use a legacy code path and ask the Python > SDK harness to execute a transform with a payload > *ref_AppliedPTransform_AppliedPTransform_45* instead of sending the > PTransform proto. > -- This message was sent by Atlassian Jira (v8.3.4#803005)