mwallace582 opened a new issue, #24470: URL: https://github.com/apache/beam/issues/24470
### What happened? Hi Apache Beam team, I've recently identified an issue with using the Beam Go SDK with Google's Dataflow. I've already [filed a bug report with Google](https://issuetracker.google.com/issues/260992167), but I wanted to report it here as well in case there's something to be done in the SDK side. I'll paste the contents of my liked bug report below: > Recently, I encountered an issue where I was unable to run any Dataflow pipeline using the Go language SDK. Previously I'd only used Python with Dataflow. I used the `wordcount.go` example pipeline as my test case. I followed the deployment instructions on the Dataflow site and didn't do anything fancy. > > Upon deploying the pipeline, the Dataflow console reported that the workers had started successfully and that everything was running fine. Despite this, the pipeline never made any progress whatsoever. > > I noticed some errors at the logs when looking at Stackdriver with these filters: > > ``` > resource.type="dataflow_step" > resource.labels.job_id="2022-11-30_13_25_07-7319605715258819834" > ``` > > The most concerning log entries were the following: > > ``` > "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 1dd1f5aed1062c9fa85e1437dcbb915a8e3844a5a8de66982aaf4c1471dc492e" containerID="1dd1f5aed1062c9fa85e1437dcbb915a8e3844a5a8de66982aaf4c1471dc492e" > "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: 1dd1f5aed1062c9fa85e1437dcbb915a8e3844a5a8de66982aaf4c1471dc492e" pod="default/df-matthew-test-golang-v3-11301257-l37l-harness-6jnd" > "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"sdk-0-0\" with CrashLoopBackOff: \"back-off 10s restarting failed container=sdk-0-0 pod=df-matthew-test-golang-v3-11301257-l37l-harness-6jnd_default(57e4bc85cfdd8d814463331df622de1a)\"" pod="default/df-matthew-test-golang-v3-11301257-l37l-harness-6jnd" podUID=57e4bc85cfdd8d814463331df622de1a > "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"sdk-0-0\" with CrashLoopBackOff: \"back-off 20s restarting failed container=sdk-0-0 pod=df-matthew-test-golang-v3-11301257-l37l-harness-6jnd_default(57e4bc85cfdd8d814463331df622de1a)\"" pod="default/df-matthew-test-golang-v3-11301257-l37l-harness-6jnd" podUID=57e4bc85cfdd8d814463331df622de1a > "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"sdk-0-0\" with CrashLoopBackOff: \"back-off 20s restarting failed container=sdk-0-0 pod=df-matthew-test-golang-v3-11301257-l37l-harness-6jnd_default(57e4bc85cfdd8d814463331df622de1a)\"" pod="default/df-matthew-test-golang-v3-11301257-l37l-harness-6jnd" podUID=57e4bc85cfdd8d814463331df622de1a > ``` > > I was stuck here for awhile, not understanding what was going wrong which could explain these Kubernetes errors. After staring at the logs for several hours, I came across these log messages, which were not marked as errors: > > ``` >2022/12/01 00:40:02 Provision info: >pipeline_options:{fields:{key:"beam:option:go_options:v1" value:{struct_value:{fields:...blah..blah}}}} >2022/12/01 00:40:02 Initializing Go harness: /opt/apache/beam/boot --logging_endpoint=localhost:12370 --control_endpoint=localhost:12371 --artifact_endpoint=localhost:12372 --provision_endpoint=localhost:12373 --semi_persist_dir=/var/opt/google --id=sdk-0-0 > /bin/worker: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /bin/worker) > /bin/worker: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /bin/worker) > 2022/12/01 00:40:02 User program exited: exit status 1 > ignoring event map[container:cc38c9371adfcaff693d3b7bb1791111f7793f6d875eea6e756bb0254433b09e module:libcontainerd namespace:moby topic:/tasks/delete type:*events.TaskDelete] > ``` > > Not being an expert on the internals of Dataflow, I didn't know if this was a real error or one of the many benign errors that appear in the Dataflow logs. But I figured that I would assume that these errors were the errors causing my issues, and see if I could quash them. > > My development machine has glibc 2.36 installed, and the `apache/beam_go_sdk:2.43.0` docker image has glibc 2.31 installed. Instead of downgrading my glibc version, I did some googling, and found that I could disable Go's dynamic linking by exporting `CGO_ENABLED=0` before deploying the pipeline with go run. This fixed the issue with my pipeline immediately. > > I'm fairly new to Go, so I can't say with any certainty that dynamic linking should be disabled by default on all Dataflow Go pipelines, but it is something worth considering. Alternatively, it would be EXTREMELY helpful if Dataflow would have reported this error more clearly in the first place. I don't think that mismatched glibc versions are a particularly rare occurrence, and this subtle error is a huge barrier to entry when using Go with Dataflow. ### Issue Priority Priority: 2 ### Issue Component Component: sdk-go -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
