phoerious commented on a change in pull request #16658:
URL: https://github.com/apache/beam/pull/16658#discussion_r810544786
##########
File path: sdks/python/container/boot.go
##########
@@ -145,7 +145,21 @@ func main() {
// Guard from concurrent artifact retrieval and installation,
// when called by child processes in a worker pool.
+ workerPoolId := os.Getenv(workerPoolIdEnv)
+ var venvDir string
+ if workerPoolId != "" {
+ venvDir = filepath.Join(*semiPersistDir, "beam-venv",
"beam-pool-" + workerPoolId)
+ } else {
+ venvDir = filepath.Join(*semiPersistDir, "beam-venv",
"beam-worker-" + *id)
Review comment:
I ran into some issues when multiple workers were spawned in the same
container. I think I have to bind the venv to the worker and then clean them up
individually. Otherwise some workers will get their venv deleted by other
workers in the same container. I will check next week how best to solve this
and update the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
