phoerious commented on issue #24357: URL: https://github.com/apache/beam/issues/24357#issuecomment-1669254788
@tvalentyn I think I may have found why this is happening. The following line connects the container executable's STDOUT and then simply starts the process without interacting with it any further. This will fill up the internal I/O buffers and hang forever unless the output is consumed somehow, e.g. by calling `communicate()`. https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/worker_pool_main.py#L167 This behaviour is documented: https://docs.python.org/3/library/subprocess.html#subprocess.Popen.wait This is a general problem with pipes, so it's not specific to Python. These downstream lines in Go are just as problematic and may already cause deadlocks if the upstream Python call doesn't: https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/util/execx/exec.go#L34-L36 See also https://devblogs.microsoft.com/oldnewthing/20110707-00/?p=10223 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
