Note that so far we have not been staging wheels, since SDK does not have a knowledge of a target platform, but there is https://issues.apache.org/jira/browse/BEAM-4032 to add this support.
On Thu, Dec 5, 2019 at 2:35 PM Chad Dombrova <chad...@gmail.com> wrote: > On Thu, Dec 5, 2019 at 12:36 PM Valentyn Tymofieiev <valen...@google.com> > wrote: > > Ah nice, so then the workflow would be: download [missing] deps from pypi >> into a long-lived cache directory, then download copy the same deps into >> a short-lived temporary directory, using long-lived cache directory as >> SoT, then stage files from a short-lived temporary directory and clean it >> up. Is that what you are suggesting, Chad? >> > Yes, I just did a quick test to confirm: > > # download or build wheels of anything that's missing from the cache > # note: we're including gcp extras: > pip wheel apache_beam[gcp]==2.16 --wheel-dir /tmp/wheel-cache > # copy some of those wheels somewhere else > # note: we're excluding gcp extras > pip download apache_beam==2.16 --no-binary --find-links=/tmp/wheel-cache > --dest /tmp/wheel-dest/ > # rerun to confirm that cached wheels are being re-used instead of downloaded > from pypi > pip wheel apache_beam[gcp]==2.16 --wheel-dir /tmp/wheel-cache > > /tmp/wheel-dest/ will now have a subset of the deps from /tmp/wheel-cache, > excluding the gcp extras. > > Note that for some reason the equal sign after —find-links is required, at > least for me on pip 19.1.1. Using a space resulted in an error. > > -chad > > >
