[
https://issues.apache.org/jira/browse/BEAM-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17510777#comment-17510777
]
Beam JIRA Bot commented on BEAM-13669:
--------------------------------------
This issue is P2 but has been unassigned without any comment for 60 days so it
has been labeled "stale-P2". If this issue is still affecting you, we care!
Please comment and remove the label. Otherwise, in 14 days the issue will be
moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed
explanation of what these priorities mean.
> Install Python wheel and dependencies to local venv in SDK harness
> ------------------------------------------------------------------
>
> Key: BEAM-13669
> URL: https://issues.apache.org/jira/browse/BEAM-13669
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-harness
> Affects Versions: 2.35.0
> Reporter: Janek Bevendorff
> Priority: P2
> Labels: stale-P2, usability
>
> With the {{--setup-file}} option, the Python SDK harness compiles and
> installs a wheel and its package dependencies on the taskmanager SDK sidecar
> containers. Unfortunately, this installation is global and persistent, so
> consecutive job submissions will reuse the previously installed wheel instead
> of reinstalling or updating it. This makes it impossible to submit updated
> code (beyond the main .py file, which will always be fresh) without deleting
> and recreating all taskmanager containers. It also messes with dependencies
> of other jobs and can cause inconsistencies or errors that are extremely hard
> to debug.
> *Suggestion:* Create a temporary venv per job, install packages there, delete
> venv after job completion or failure.
> *Fixes:* Stale or conflicting dependencies in persistent SDK containers. Pip
> will not reinstall existing packages by default, even if their version
> numbers differ and the number of installed dependency packages grows
> unnecessarily over time. There is also no point in keeping the
> {{--setup-file}} wheel installed for all eternity.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
