[
https://issues.apache.org/jira/browse/BEAM-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17479568#comment-17479568
]
Kenneth Knowles commented on BEAM-13669:
----------------------------------------
I am not certain what deployment you are using here. I believe the expectation
is that SDK harness/sidecar containers are tied to a specific job and typically
ephemeral. Beam has a concept of "environment" and it is not really safe to
re-use across jobs.
> Install Python wheel and dependencies to local venv in SDK harness
> ------------------------------------------------------------------
>
> Key: BEAM-13669
> URL: https://issues.apache.org/jira/browse/BEAM-13669
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-harness
> Affects Versions: 2.35.0
> Reporter: Janek Bevendorff
> Priority: P2
>
> With the {{--setup-file}} option, the Python SDK harness compiles and
> installs a wheel and its package dependencies on the taskmanager SDK sidecar
> containers. Unfortunately, this installation is global and persistent, so
> consecutive job submissions will reuse the previously installed wheel instead
> of reinstalling or updating it. This makes it impossible to submit updated
> code (beyond the main .py file, which will always be fresh) without deleting
> and recreating all taskmanager containers. It also messes with dependencies
> of other jobs and can cause inconsistencies or errors that are extremely hard
> to debug.
> *Suggestion:* Create a temporary venv per job, install packages there, delete
> venv after job completion or failure.
> *Fixes:* Stale or conflicting dependencies in persistent SDK containers. Pip
> will not reinstall existing packages by default, even if their version
> numbers differ and the number of installed dependency packages grows
> unnecessarily over time. There is also no point in keeping the
> {{--setup-file}} wheel installed for all eternity.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
