[
https://issues.apache.org/jira/browse/BEAM-4032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17167770#comment-17167770
]
Kris Wilson commented on BEAM-4032:
-----------------------------------
this is blocking some experimentation-focused usages of Beam (and TFX) in the
computational notebook environment at Twitter, so we'd love to get this
re-prioritized. This will continue to be a problem for us even beyond custom
container support, because we'll still want a way to get dynamic dependencies
onto workers faster at the Python environment layer vs the heavier-weight OS
container layer (which has a much higher round-trip image build + push cost
than e.g. pip install).
we'd love to contribute and have significant experience with Python dependency
management and cross-platform python package resolution - we'll circle back to
dig in deeper on the contrib process and code base, but in the meantime if
anyone has any pointers or other ideas here we're all ears.
> Support staging binary distributions of dependency packages.
> ------------------------------------------------------------
>
> Key: BEAM-4032
> URL: https://issues.apache.org/jira/browse/BEAM-4032
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
> Reporter: Valentyn Tymofieiev
> Priority: P3
>
> requirements.txt only supports source-distribution dependencies [1].
> --extra_packages does not officially support wheel files [2].
> It is possible to expand this to support binary distributions as long as we
> have the knowledge of the target platform.
> We should take into consideration the mechanisms of staging dependencies
> through portability framework, and perhaps consolidate some of the existing
> options.
> [https://github.com/apache/beam/blob/a79d1b4fc27eb81db0d9a773047820a206f3d238/sdks/python/apache_beam/runners/dataflow/internal/dependency.py#L260]
> [https://github.com/apache/beam/blob/a79d1b4fc27eb81db0d9a773047820a206f3d238/sdks/python/apache_beam/runners/dataflow/internal/dependency.py#L188]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
