Hi everyone, Austin Benett I would like cooperate with you with introducing GitHub actions for python wheels.
Since last week I am trying to figure out how to do this and I was able to build wheels for Linux and MacOS with help of https://github.com/joerick/cibuildwheel/ <https://github.com/joerick/cibuildwheel/>. I’ve created draft PR to apache/beam: https://github.com/apache/beam/pull/11877 <https://github.com/apache/beam/pull/11877> and you can check GitHub actions logs on fork PR here: https://github.com/TobKed/beam/pull/3 <https://github.com/TobKed/beam/pull/3> I am struggling with building wheels for Windows due to the error: > Cannot open include file: 'unistd.h': No such file or directory caused by file sdks/python/apache_beam/runners/worker/statesampler_fast.pyx Adding this file to be excluded by cythonize method in setup.py allowed build to be completed however I don’t think it is a solution. Mentioned error on GitHub actions: https://github.com/TobKed/beam/runs/727709583?check_suite_focus=true#step:7:977 <https://github.com/TobKed/beam/runs/727709583?check_suite_focus=true#step:7:977> and my local PR: https://github.com/TobKed/beam/pull/2 <https://github.com/TobKed/beam/pull/2> What do you think about it? Tobiasz On 2020/05/30 20:53:51, Austin Bennett <[email protected]> wrote: > There seems to be support here for this idea. After digging through> > things, I *think* I understand the moving pieces and can address (reason I> > was digging through the code, including beam-wheels repo [1]). Though, not> > 100% that I have not overlooked bits.> > > Would be happy to pickup> > https://issues.apache.org/jira/browse/BEAM-9388 (imagine> > would get to it over the course of a month or few -- so also wouldn't want> > to take it on if others would be looking to do it sooner) -- seems> > worthwhile to reduce manual steps for the release process, as well as the> > additional implications (daily builds) if can easily manage. I have some> > experience with GitHub Actions, but not with our (BEAM) deployment/release> > process, nor much with building wheels. I assume there are sufficient docs> > available to go through for the mechanics of the latter. If picking up> > this issue, I suspect may need to bother some of those that have done> > recent releases -- that would best occur on the jira ticket, once at that> > point?> > > @Ismail <[email protected]>-- given your experience with GitHub actions,> > any reason to think GitHub Actions not appropriate/ready for this specific> > task?> > > @Ahmet Altay <[email protected]> happy to understand the extent of what you> > had in mind, maybe the extensions are not as important to plan out, as> > they're straightforwardly bolted on (ex: daily builds). More tactically> > would be valuable to ensure I understand what all needs to occur. Any> > other source of info to consume other than> > https://github.com/apache/beam-wheels and> > https://beam.apache.org/contribute/release-guide/.> > > Also, open to the thought that this might be taking on too much, without> > more experience with the release process and such. Do advise...?> > > > [1]> > https://lists.apache.org/thread.html/r93884eb080297647207f7d2b8a393e224029fc2c3509017886e84051%40%3Cdev.beam.apache.org%3E> > > > > > ---------- Forwarded message ---------> > From: Ismaël Mejía <[email protected]>> > Date: Wed, Feb 26, 2020 at 12:09 PM> > Subject: Re: [DISCUSSION] Use github actions for python wheels ?> > To: dev <[email protected]>> > > > +1 I have been migrating multiple projects into github actions recently and> > even if for some tasks it is not as mature and polished as travis it has> > proven to be way more reliable.> > > > On Wed, Feb 26, 2020 at 7:07 PM Ahmet Altay <[email protected]> wrote:> > > > I created https://issues.apache.org/jira/browse/BEAM-9388 to explore> > > this. To be explicit and not to do cookie licking, I would not be able to> > > work on this at the moment. If anyone is interested please take it.> > > Otherwise I will try to come back and explore this when I can.> > >> > > On Tue, Feb 25, 2020 at 2:57 PM Robert Bradshaw <[email protected]>> > > wrote:> > >> > >> I'd be in favor of this, assuming it actually simplifies things.> > >> > >> > > This is also my concern. I do think that it will simplify things, but I am> > > not certain as I am not very familiar with the github actions.> > >> > >> > >> (Note> > >> that the wheels are for several variants of linux, presumably we could> > >> do cross-compiles. Also, manylinux is a "minimal" linux specifically> > >> built as to produce shared object libraries compatible with a wide> > >> variety of distributions--we can't just assume that a shared object> > >> library built on one modern linux will just work on another. (But> > >> maybe it's sufficient to do this within a docker environment?)> > >>> > >> > > There will be no change in this area. Both in Both Travis and github> > > actions offer a comparable set of options.> > >> > >> > >>> > >> On Tue, Feb 25, 2020 at 2:23 PM Kenneth Knowles <[email protected]> wrote:> > >> >> > >> > +1 to exploring this.> > >> >> > >> > On [email protected] there is lots of discussion and general approval> > >> for trying it. It is enabled and used by some projects. Calcite uses it > >> to> > >> build their website, for example.> > >>> > >> > > Great.> > >> > >> > >> >> > >> > Kenn> > >> >> > >> >> > >> > On Tue, Feb 25, 2020 at 2:08 PM Ahmet Altay <[email protected]> wrote:> > >> >>> > >> >> Hi all,> > >> >>> > >> >> I recently had a chance to look at the documentation for github> > >> actions. I think we could use github actions instead of travis to for> > >> building python wheels during releases. This will have the following> > >> advantages:> > >> >>> > >> >> - We will eliminate one repo. (If you don't know, we have> > >> https://github.com/apache/beam-wheels for the sole purpose of building> > >> wheels file.)> > >> >> - Workflow will be stored in the same repo. This will prevent bit rot> > >> that is only discovered at release times. (happened a few times, although> > >> usually easy to fix.)> > >> >> - github actions supports ubuntu, mac, windows environments. We could> > >> try to build wheels for windows as well. (Travis also supports the same> > >> environments but we only use linux and mac environments. Maybe there are> > >> other blockers for building wheels for Windows.)> > >> >> - We could do more, like daily python builds.> > >> >>> > >> >> Downsides would be:> > >> >> - I do not know if github actions will require some special set of> > >> permissions that require an approval from infra.> > >> >> - Travis works fine most of the time. This might be unnecessary work.> > >> >>> > >> >> What do you think? Is this feasible, would this be useful?> > >> >>> > >> >> Ahmet> > >>> > >> >
