phoerious opened a new issue, #29663: URL: https://github.com/apache/beam/issues/29663
### What would you like to happen? In #16658, I added the feature that the Python SDK harness `boot` binary installs all application dependencies into a temporary venv to make containers reusable. At the moment, this is hard-coded to /opt/apache/beam-venv and falls back to the default env if that fails or if `RUN_PYTHON_SDK_IN_DEFAULT_ENVIRONMENT` is set. Now my own requirements have shifted and I would like to make this path configurable. In a first iteration of my PR back then, I used the value of the `-semiPersistDir` flag, which defaults to `/tmp`. Unfortunately, this caused some Dataflow tests to fail (iirc), so we decided to use /opt/apache/beam-venv, which is writable by the beam user in the upstream Python SDK container, but may not be so in other environments. I think, using /tmp would make sense, but making it configurable would be better. Even better would be if the runner implementation could set this automatically. For instance, the Spark runner would set this to `spark.local.dir`. I could submit a PR to make this configurable with a flag (either -semiPersistDir or an additional flag if this still doesn't work). I would not know, however, how to get this value automatically from the job runner. Perhaps someone else has an idea? cc @tvalentyn ### Issue Priority Priority: 2 (default / most feature requests should be filed as P2) ### Issue Components - [X] Component: Python SDK - [ ] Component: Java SDK - [ ] Component: Go SDK - [ ] Component: Typescript SDK - [ ] Component: IO connector - [ ] Component: Beam YAML - [ ] Component: Beam examples - [ ] Component: Beam playground - [ ] Component: Beam katas - [ ] Component: Website - [ ] Component: Spark Runner - [ ] Component: Flink Runner - [ ] Component: Samza Runner - [ ] Component: Twister2 Runner - [ ] Component: Hazelcast Jet Runner - [ ] Component: Google Cloud Dataflow Runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
