I think the thing is just that you can point a user env at it to test like a user. Like you can do this with https://repository.apache.org/content/repositories/orgapachebeam-1030/ but not https://dist.apache.org/repos/dist/dev/beam/2.4.0/
On Wed, Mar 7, 2018 at 5:03 PM Ahmet Altay <al...@google.com> wrote: > I do not know what is the best practice. For practical purposes it makes > sense to stage to the same svn repo, so that we can test it as part of the > release process. > > On Wed, Mar 7, 2018 at 4:22 PM, Robert Bradshaw <rober...@google.com> > wrote: > >> Yes, we should. There's a bit of an open question of where these release >> artifacts should be staged. (Eventually, of course, they'll be published to >> PyPi). Should they be placed alongside the source artifacts in the svn >> repository? >> >> >> On Wed, Mar 7, 2018 at 3:00 PM Ahmet Altay <al...@google.com> wrote: >> >>> Are we planning to do this for the 2.4.0 release? I am asking, because >>> they were not part of RC1 artifacts. >>> >>> On Tue, Feb 13, 2018 at 9:18 AM, Robert Bradshaw <rober...@google.com> >>> wrote: >>> >>>> On Tue, Feb 13, 2018 at 8:31 AM, Nima Mousavi <nima.mous...@gmail.com> >>>> wrote: >>>> > Related question: >>>> > >>>> > How can we tell if the docker image of our binary contains the cython >>>> > optimized beam or the slower codepath? >>>> > The image was built on Google cloud (using gcloud container builds >>>> submit). >>>> >>>> There are certain modules (corresponding to the pyx files) that are >>>> only built if Cython is present. We can (1) make sure Cython is >>>> installed before installing apache beam into the container, and (2) >>>> assert as part of the build process that these modules exist. >>>> >>>> > On Mon, Feb 12, 2018 at 9:32 PM, Ahmet Altay <al...@google.com> >>>> wrote: >>>> >> >>>> >> +1 to wheels. The main effort for this would be updating the release >>>> >> guide, and adding support for other platforms in Jenkins for >>>> building and >>>> >> testing wheels. In light of this, maybe we can prioritize having >>>> test >>>> >> infrastructure for other platforms. >>>> >> >>>> >> On Mon, Feb 12, 2018 at 1:47 PM, Ismaël Mejía <ieme...@gmail.com> >>>> wrote: >>>> >>> >>>> >>> +1 for wheels, they are the standard binary distribution format so >>>> it >>>> >>> makes sense. Also wheels support packaging python 2 and 3 on >>>> universal >>>> >>> packages so they are future proof. >>>> >>> >>>> >>> On Mon, Feb 12, 2018 at 10:26 PM, Robert Bradshaw < >>>> rober...@google.com> >>>> >>> wrote: >>>> >>> > +1, is it too late to try to release these as part of the 2.3 >>>> release >>>> >>> > (to get familiar with the process, no code changes should be >>>> needed)? >>>> >> >>>> >> >>>> >> It would be nice to have this for the current release. How can we >>>> build >>>> >> and test these binaries? I think it will be prudent to waIt until we >>>> have >>>> >> infrastructure. >>>> >> >>>> >>> >>>> >>> > >>>> >>> > The wheels are advantageous when running locally (e.g. during >>>> testing >>>> >>> > and development) where requiring containers will probably be >>>> overkill. >>>> >>> > This will become especially relevant with the switch to use the >>>> >>> > FnApiRunner. >>>> >>> > >>>> >>> > On Mon, Feb 12, 2018 at 1:22 PM, Lukasz Cwik <lc...@google.com> >>>> wrote: >>>> >>> >> If we want all our code related to pipeline execution to be in a >>>> >>> >> container, >>>> >>> >> what value does building wheel distributions provide? >>>> >>> >> >>>> >>> >> >>>> >>> >> On Mon, Feb 12, 2018 at 1:18 PM, Kenneth Knowles <k...@google.com >>>> > >>>> >>> >> wrote: >>>> >>> >>> >>>> >>> >>> +1 >>>> >>> >>> >>>> >>> >>> On Mon, Feb 12, 2018 at 1:04 PM, Charles Chen <c...@google.com> >>>> wrote: >>>> >>> >>>> >>>> >>> >>>> Currently, Apache Beam distributes Python packages through pip >>>> and >>>> >>> >>>> PyPI. >>>> >>> >>>> On PyPI, developers can release either source tarballs, and / >>>> or >>>> >>> >>>> precompiled >>>> >>> >>>> "wheel" distributions for each platform, which would be used if >>>> >>> >>>> available >>>> >>> >>>> for a particular platform. Currently, we only distribute the >>>> source >>>> >>> >>>> tarballs, so any user who installs Beam using "pip install >>>> >>> >>>> apache_beam" has >>>> >>> >>>> to have a compiler and toolchain installed to take advantage of >>>> >>> >>>> Cython >>>> >>> >>>> optimizations in Beam (which require compiled C code). If >>>> such a >>>> >>> >>>> compiler >>>> >>> >>>> is not available, Beam is currently configured to install >>>> anyway, >>>> >>> >>>> but will >>>> >>> >>>> use slower Python codepaths instead of the more optimized ones >>>> (for >>>> >>> >>>> example, >>>> >>> >>>> for Coder encoding / decoding). >>>> >>> >>>> >>>> >>> >>>> I would like to propose that we start distributing binary wheel >>>> >>> >>>> distributions for our releases, for common platforms like >>>> Windows / >>>> >>> >>>> Mac / >>>> >>> >>>> Linux. We could potentially use a method similar to this one >>>> >>> >>>> (https://github.com/MacPython/cython-wheels) for building >>>> these >>>> >>> >>>> wheel >>>> >>> >>>> distributions. Thoughts? >>>> >>> >>>> >>>> >>> >>>> Best, >>>> >>> >>>> Charles >>>> >>> >>> >>>> >>> >>> >>>> >>> >> >>>> >> >>>> >> >>>> > >>>> >>> >>> >
