[
https://issues.apache.org/jira/browse/BEAM-4032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17168181#comment-17168181
]
Valentyn Tymofieiev commented on BEAM-4032:
-------------------------------------------
To make this work we need to identify which binary distribution to stage. This
depends on the target platform on the execution environment/runner.
Dependency staging happens in the SDK[1]. Installation of the packages happens
in runners, for example [2].
Which packages are compatible with the platform should be defined by PEP-0425
[3], we can also look for how the package selection logic is implemented in pip.
https://github.com/apache/beam/blob/010adc5e8640035cf4661e00bf2ed019ae27f2ce/sdks/python/apache_beam/runners/portability/stager.py#L537
https://github.com/apache/beam/blob/010adc5e8640035cf4661e00bf2ed019ae27f2ce/sdks/python/container/boot.go#L223
https://www.python.org/dev/peps/pep-0425/
> Support staging binary distributions of dependency packages.
> ------------------------------------------------------------
>
> Key: BEAM-4032
> URL: https://issues.apache.org/jira/browse/BEAM-4032
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
> Reporter: Valentyn Tymofieiev
> Priority: P3
>
> requirements.txt only supports source-distribution dependencies [1].
> --extra_packages does not officially support wheel files [2].
> It is possible to expand this to support binary distributions as long as we
> have the knowledge of the target platform.
> We should take into consideration the mechanisms of staging dependencies
> through portability framework, and perhaps consolidate some of the existing
> options.
> [https://github.com/apache/beam/blob/a79d1b4fc27eb81db0d9a773047820a206f3d238/sdks/python/apache_beam/runners/dataflow/internal/dependency.py#L260]
> [https://github.com/apache/beam/blob/a79d1b4fc27eb81db0d9a773047820a206f3d238/sdks/python/apache_beam/runners/dataflow/internal/dependency.py#L188]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
