ronoaldo opened a new issue, #22931: URL: https://github.com/apache/beam/issues/22931
### What happened? Following the documentation available [here](https://beam.apache.org/documentation/programming-guide/#1323-using-cross-language-transforms-in-a-go-pipeline) for Beam 2.4.1, I am trying to write a simple pipeline in Go using an external transform defined in Python. I followed the docs with my own "word count" code but ended up having an error that was very obscure. Then I decided to run the Python test cross lang suite (pip installed at `apache_beam/runners/portability/expansion_service_test.py`) to use it as expansion service entrypoint and call a sample from the Go SDK tree, specifically, the xlang/wordcount sample. To my surprise, It failed with the same kind of error. After several trial and errors I failed to get Go pipeline call an external Python transform following the docs and even reading some code from the stack traces. Here is the output from the Python expansion service: ``` (env) $ python ../env/lib/python3.9/site-packages/apache_beam/runners/portability/expansion_service_test.py -p 12345 INFO:apache_beam.runners.portability.stager:Executing command: ['/home/ronoaldo/workspace/micro-beam/env/bin/python', '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/tmp/tmpomfse_4t/tmp_requirements.txt', '--exists-action', 'i', '--no-deps', '--implementation', 'cp', '--abi', 'cp39', '--platform', 'manylinux2014_x86_64'] INFO:root:Default Python SDK image for environment is apache/beam_python3.9_sdk:2.41.0 INFO:root:No image given, using default Python SDK image INFO:root:Default Python SDK image for environment is apache/beam_python3.9_sdk:2.41.0 INFO:root:Python SDK container image set to "apache/beam_python3.9_sdk:2.41.0" for Docker environment INFO:__main__:Listening for expansion requests at 12345 INFO:root:Missing pipeline option (runner). Executing pipeline using the default runner: DirectRunner. ``` This is the error from the pipeline (very similar one I got with my own code, except it was complaining for 'n5' but same KeyError at the same place): ``` (env) $ wordcount --expansion_addr=localhost:12345 panic: tried cross-language for beam:transforms:xlang:count against localhost:12345 and failed expanding external transform expanding transform with ExpansionRequest: components:{pcollections:{key:"n3" value:{unique_name:"n3" coder_id:"c0@OfMQSsncEv" is_bounded:BOUNDED windowing_strategy_id:"w0@OfMQSsncEv"}} windowing_strategies:{key:"w0@OfMQSsncEv" value:{window_fn:{urn:"beam:window_fn:global_windows:v1"} merge_status:NON_MERGING window_coder_id:"c1@OfMQSsncEv" trigger:{default:{}} accumulation_mode:DISCARDING output_time:END_OF_WINDOW closing_behavior:EMIT_IF_NONEMPTY on_time_behavior:FIRE_IF_NONEMPTY environment_id:"go"}} coders:{key:"c0@OfMQSsncEv" value:{spec:{urn:"beam:coder:string_utf8:v1"}}} coders:{key:"c1@OfMQSsncEv" value:{spec:{urn:"beam:coder:global_window:v1"}}} environments:{key:"go" value:{}}} transform:{unique_name:"External" spec:{urn:"beam:transforms:xlang:count"} inputs:{key:"i0" value:"n3"} environment_id:"go"} namespace:"OfMQSsncEv" expansion failed caused by: Traceback (most recent call last): File "/home/ronoaldo/workspace/micro-beam/env/lib/python3.9/site-packages/apache_beam/runners/portability/expansion_service.py", line 77, in Expand inputs = transform._pvaluish_from_dict({ File "/home/ronoaldo/workspace/micro-beam/env/lib/python3.9/site-packages/apache_beam/runners/portability/expansion_service.py", line 79, in <dictcomp> with_pipeline(context.pcollections.get_by_id(pcoll_id), pcoll_id) File "/home/ronoaldo/workspace/micro-beam/env/lib/python3.9/site-packages/apache_beam/runners/portability/expansion_service.py", line 49, in with_pipeline component.producer, component.tag = producers[pcoll_id] KeyError: 'n3' goroutine 1 [running]: github.com/apache/beam/sdks/v2/go/pkg/beam.CrossLanguage({0xc000480ac0?, 0xc0004b1090?}, {0xfdbd06, 0x1b}, {0x0?, 0x8?, 0x7f39103faa68?}, {0x7fffd9812cdc, 0xf}, 0xc0006bfc68, ...) /home/ronoaldo/go/pkg/mod/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/xlang.go:162 +0x136 github.com/apache/beam/sdks/v2/go/examples/xlang.Count({0xc0004809c0?, 0xc0004b1090?}, {0x7fffd9812cdc, 0xf}, {0x0?}) /home/ronoaldo/go/pkg/mod/github.com/apache/beam/sdks/[email protected]/go/examples/xlang/transforms.go:113 +0x3d4 main.main() /home/ronoaldo/go/pkg/mod/github.com/apache/beam/sdks/[email protected]/go/examples/xlang/wordcount/wordcount.go:94 +0x22e ``` I tested this using a Python virtual env with `apache-beam[gcp]` and `go install`'ed this example: `github.com/apache/beam/sdks/v2/go/examples/xlang/wordcount@latest`. From one terminal session I launched the expansion service and on another the wordcount Go program. ### Issue Priority Priority: 2 ### Issue Component Component: sdk-go -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
