ronoaldo commented on issue #29180: URL: https://github.com/apache/beam/issues/29180#issuecomment-1882046847
I'm facing a similar issue posted by the previous comment by @johannaojeling in this sample pipeline: https://github.com/ronoaldo/micro-beam/blob/d074d5fe72b32d36e84c1007ae68efebe13a3821/05_xlang/go/pipeline.go#L126 It's derived from the minimal wordcount and changed to allow for xlang transform execution demostration. However, it is also able to run as a "standalone" pipeline if no expansion server is provided using just Go pipeline code. Without any expansion server (i.e., just Go code), it works fine under DirectRunner or DataflowRunner, and does not work on Prism with the stalling loop reading from the TextIO: ``` requirements: "beam:requirement:pardo:splittable_dofn:v1" 2024/01/08 21:21:44 Prepared job with id: job-001 and staging token: job-001 2024/01/08 21:21:44 Staged binary artifact with token: job-001 2024/01/08 21:21:44 Submitted job: job-001 2024/01/08 21:21:44 (): starting job-001[go-job-1-1704759704240538359] 2024/01/08 21:21:44 (): running job-001[go-job-1-1704759704240538359] 2024/01/08 21:21:44 Job[job-001] state: RUNNING 2024/01/08 21:21:44 starting worker job-001[go-job-1-1704759704240538359]_go 2024/01/08 21:21:45 INFO Reading from gs://dataflow-samples/shakespeare/kinglear.txt source=/home/ronoaldo/go/pkg/mod/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/textio/textio.go:226 time=2024-01-09T00:21:45.075Z worker.ID=job-001[go-job-1-1704759704240538359]_go worker.endpoint=localhost:46495 2024/01/08 21:21:45 INFO Reading from gs://dataflow-samples/shakespeare/kinglear.txt source=/home/ronoaldo/go/pkg/mod/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/textio/textio.go:226 time=2024-01-09T00:21:45.376Z worker.ID=job-001[go-job-1-1704759704240538359]_go worker.endpoint=localhost:46495 2024/01/08 21:21:45 INFO Reading from gs://dataflow-samples/shakespeare/kinglear.txt source=/home/ronoaldo/go/pkg/mod/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/textio/textio.go:226 time=2024-01-09T00:21:45.476Z worker.ID=job-001[go-job-1-1704759704240538359]_go worker.endpoint=localhost:46495 2024/01/08 21:21:45 INFO Reading from gs://dataflow-samples/shakespeare/kinglear.txt source=/home/ronoaldo/go/pkg/mod/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/textio/textio.go:226 time=2024-01-09T00:21:45.576Z worker.ID=job-001[go-job-1-1704759704240538359]_go worker.endpoint=localhost:46495 2024/01/08 21:21:45 INFO Reading from gs://dataflow-samples/shakespeare/kinglear.txt source=/home/ronoaldo/go/pkg/mod/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/textio/textio.go:226 time=2024-01-09T00:21:45.677Z worker.ID=job-001[go-job-1-1704759704240538359]_go worker.endpoint=localhost:46495 2024/01/08 21:21:45 INFO Reading from gs://dataflow-samples/shakespeare/kinglear.txt source=/home/ronoaldo/go/pkg/mod/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/textio/textio.go:226 time=2024-01-09T00:21:45.677Z worker.ID=job-001[go-job-1-1704759704240538359]_go worker.endpoint=localhost:46495 2024/01/08 21:21:45 INFO Reading from gs://dataflow-samples/shakespeare/kinglear.txt source=/home/ronoaldo/go/pkg/mod/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/textio/textio.go:226 time=2024-01-09T00:21:45.778Z worker.ID=job-001[go-job-1-1704759704240538359]_go worker.endpoint=localhost:46495 2024/01/08 21:21:45 INFO Reading from gs://dataflow-samples/shakespeare/kinglear.txt source=/home/ronoaldo/go/pkg/mod/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/textio/textio.go:226 time=2024-01-09T00:21:45.778Z worker.ID=job-001[go-job-1-1704759704240538359]_go worker.endpoint=localhost:46495 2024/01/08 21:21:45 INFO Reading from gs://dataflow-samples/shakespeare/kinglear.txt source=/home/ronoaldo/go/pkg/mod/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/textio/textio.go:226 time=2024-01-09T00:21:45.778Z worker.ID=job-001[go-job-1-1704759704240538359]_go worker.endpoint=localhost:46495 2024/01/08 21:21:45 INFO Reading from gs://dataflow-samples/shakespeare/kinglear.txt source=/home/ronoaldo/go/pkg/mod/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/textio/textio.go:226 time=2024-01-09T00:21:45.922Z worker.ID=job-001[go-job-1-1704759704240538359]_go worker.endpoint=localhost:46495 2024/01/08 21:21:45 INFO Reading from gs://dataflow-samples/shakespeare/kinglear.txt source=/home/ronoaldo/go/pkg/mod/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/textio/textio.go:226 time=2024-01-09T00:21:45.991Z worker.ID=job-001[go-job-1-1704759704240538359]_go worker.endpoint=localhost:46495 2024/01/08 21:21:46 INFO Reading from gs://dataflow-samples/shakespeare/kinglear.txt source=/home/ronoaldo/go/pkg/mod/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/textio/textio.go:226 time=2024-01-09T00:21:46.223Z worker.ID=job-001[go-job-1-1704759704240538359]_go worker.endpoint=localhost:46495 2024/01/08 21:21:46 INFO Reading from gs://dataflow-samples/shakespeare/kinglear.txt source=/home/ronoaldo/go/pkg/mod/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/textio/textio.go:226 time=2024-01-09T00:21:46.262Z worker.ID=job-001[go-job-1-1704759704240538359]_go worker.endpoint=localhost:46495 2024/01/08 21:21:46 INFO Reading from gs://dataflow-samples/shakespeare/kinglear.txt source=/home/ronoaldo/go/pkg/mod/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/textio/textio.go:226 time=2024-01-09T00:21:46.361Z worker.ID=job-001[go-job-1-1704759704240538359]_go worker.endpoint=localhost:46495 2024/01/08 21:21:46 INFO Reading from gs://dataflow-samples/shakespeare/kinglear.txt source=/home/ronoaldo/go/pkg/mod/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/textio/textio.go:226 time=2024-01-09T00:21:46.363Z worker.ID=job-001[go-job-1-1704759704240538359]_go worker.endpoint=localhost:46495 2024/01/08 21:21:46 INFO Reading from gs://dataflow-samples/shakespeare/kinglear.txt source=/home/ronoaldo/go/pkg/mod/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/textio/textio.go:226 time=2024-01-09T00:21:46.459Z worker.ID=job-001[go-job-1-1704759704240538359]_go worker.endpoint=localhost:46495 2024/01/08 21:21:46 INFO Reading from gs://dataflow-samples/shakespeare/kinglear.txt source=/home/ronoaldo/go/pkg/mod/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/textio/textio.go:226 time=2024-01-09T00:21:46.484Z worker.ID=job-001[go-job-1-1704759704240538359]_go worker.endpoint=localhost:46495 2024/01/08 21:21:46 INFO Reading from gs://dataflow-samples/shakespeare/kinglear.txt source=/home/ronoaldo/go/pkg/mod/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/textio/textio.go:226 time=2024-01-09T00:21:46.537Z worker.ID=job-001[go-job-1-1704759704240538359]_go worker.endpoint=localhost:46495 2024/01/08 21:21:46 INFO Reading from gs://dataflow-samples/shakespeare/kinglear.txt source=/home/ronoaldo/go/pkg/mod/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/textio/textio.go:226 time=2024-01-09T00:21:46.697Z worker.ID=job-001[go-job-1-1704759704240538359]_go worker.endpoint=localhost:46495 2024/01/08 21:21:46 INFO Reading from gs://dataflow-samples/shakespeare/kinglear.txt source=/home/ronoaldo/go/pkg/mod/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/textio/textio.go:226 time=2024-01-09T00:21:46.883Z worker.ID=job-001[go-job-1-1704759704240538359]_go worker.endpoint=localhost:46495 ^Csignal: interrupt ``` It does not seems to complete the pipeline regardless of how much it is left running. I have tried to use register.Function2x1 in order to possibly fix it but it remains consistently failing when executing with Prism. Not sure if this is caused by unregistered functions but the symptom is the same. Should I open a separate issue for tracking this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
