[
https://issues.apache.org/jira/browse/BEAM-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anonymous updated BEAM-13669:
-----------------------------
Status: Triage Needed (was: In Progress)
> Install Python wheel and dependencies to local venv in SDK harness
> ------------------------------------------------------------------
>
> Key: BEAM-13669
> URL: https://issues.apache.org/jira/browse/BEAM-13669
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-harness
> Affects Versions: 2.35.0
> Reporter: Janek Bevendorff
> Priority: P1
> Labels: usability
>
> With the {{--setup-file}} option, the Python SDK harness compiles and
> installs a wheel and its package dependencies on the taskmanager SDK sidecar
> containers. Unfortunately, this installation is global and persistent, so
> consecutive job submissions will reuse the previously installed wheel instead
> of reinstalling or updating it. This makes it impossible to submit updated
> code (beyond the main .py file, which will always be fresh) without deleting
> and recreating all taskmanager containers. It also messes with dependencies
> of other jobs and can cause inconsistencies or errors that are extremely hard
> to debug.
> *Suggestion:* Create a temporary venv per job, install packages there, delete
> venv after job completion or failure.
> *Fixes:* Stale or conflicting dependencies in persistent SDK containers. Pip
> will not reinstall existing packages by default, even if their version
> numbers differ and the number of installed dependency packages grows
> unnecessarily over time. There is also no point in keeping the
> {{--setup-file}} wheel installed for all eternity.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
