Re: Apache Beam 2.4.0 release process retrospective and automation possibilities

2018-03-23 9:52 GMT+01:00 Robert Bradshaw : > To put this in context, this was a brain dump of some of the things I > encountered while doing the release. Were I to do a release again, it would > be a lot easier (though still not ideal). > > At the high level, rather than

Re: Apache Beam 2.4.0 release process retrospective and automation possibilities

To put this in context, this was a brain dump of some of the things I encountered while doing the release. Were I to do a release again, it would be a lot easier (though still not ideal). At the high level, rather than focusing on steps, I think it's more interesting to focus on why we need a

Re: Apache Beam 2.4.0 release process retrospective and automation possibilities

Hi Alan, some feedback inline: > * Several steps have high latency; it may take 30min of work before > prompting > for a password. Using GPG agent or in .m2/settings.xml works fine. It's what I'm doing in all Apache releases I'm doing. > * Problems with both his laptop and desktop;

Re: Apache Beam 2.4.0 release process retrospective and automation possibilities

Hi Le 23 mars 2018 04:29, "Alan Myrvold" a écrit : Robert explained his experience with the release process as the release engineer for 2.4.0, and we discussed the prototype shell script for checking release progress in

Re: [ANNOUNCE] Apache Beam 2.4.0 released

/rmannibucau> | Book <https://www.packtpub.com/application-development/java-ee-8-high-performance> 2018-03-22 9:50 GMT+01:00 Etienne Chauchot <echauc...@apache.org>: > Great ! > Le jeudi 22 mars 2018 à 08:24 +, Robert Bradshaw a écrit : > > We are pleased to announce the release of A

Re: [ANNOUNCE] Apache Beam 2.4.0 released

Great ! Le jeudi 22 mars 2018 à 08:24 +, Robert Bradshaw a écrit : > We are pleased to announce the release of Apache Beam 2.4.0. Thanks goes to > the many people who made this possible. > > Apache Beam is an open source unified programming model to define and > execute

[ANNOUNCE] Apache Beam 2.4.0 released

We are pleased to announce the release of Apache Beam 2.4.0. Thanks goes to the many people who made this possible. Apache Beam is an open source unified programming model to define and execute data processing pipelines, including ETL, batch and stream (continuous) processing. See https

Re: Beam 2.4.0

Done, thanks. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book

Re: Beam 2.4.0

Fully agree. If nobody jumps on it, I will tackle that tomorrow. Regards JB Le 14 mars 2018 à 18:03, à 18:03, Reuven Lax a écrit: >Can you add a description and assign a reviewer (using R:). I think >this >was basically ready to merge before modulo breaking compilation. > >

Re: Beam 2.4.0

Can you add a description and assign a reviewer (using R:). I think this was basically ready to merge before modulo breaking compilation. On Wed, Mar 14, 2018 at 10:01 AM Romain Manni-Bucau wrote: > up, know it missed the 2.4 but can >

Re: Beam 2.4.0

up, know it missed the 2.4 but can https://github.com/apache/beam/pull/4790 have some love? it really makes beam pretty unusable with direct runner, I start to have "// workaround for BEAM-3409" everywhere in my codebase which is quite bothering after 3 months :( Romain Manni-Bucau @rmannibucau

Re: Beam 2.4.0

Hi guys, tried to reapply the waitUntilFinish fix - without breaking the compilation this time ;) - in https://github.com/apache/beam/pull/4790, anyone able to help to review and move that forward? Romain Manni-Bucau @rmannibucau | Blog

Re: Beam 2.4.0

On Fri, Mar 2, 2018 at 12:01 AM Jean-Baptiste Onofré wrote: > Hi Ismaël, > > that's a good idea to show history. > > For me, the vote duration doesn't matter as we are in the release process > already. > A more relevant duration to track would probably be cut to final

Re: Beam 2.4.0

Hi Ismaël, that's a good idea to show history. For me, the vote duration doesn't matter as we are in the release process already. The gap between two releases is more significant. And clearly with an average of 80 days (~ 3 months) it's two long. The idea is to reduce this clearly. I propose

Re: Beam 2.4.0

The features/bugfixes included in a release are determined by the time between cutting release branches. So I'd focus on that cadence (outside of special requests). Kenn On Thu, Mar 1, 2018 at 12:33 PM, Robert Bradshaw wrote: > On Thu, Mar 1, 2018 at 9:17 AM Ismaël Mejía

Re: Beam 2.4.0

On Thu, Mar 1, 2018 at 9:17 AM Ismaël Mejía wrote: > The average time just in the vote process for Beam since we are out of the > incubator is 17.5 days with an average of 75 days between versions. > Good thought to look at history. I think there's general consensus that this

Re: Beam 2.4.0

The average time just in the vote process for Beam since we are out of the incubator is 17.5 days with an average of 75 days between versions. Version Vote Period No. days 2.3.030/01-16/02 17 days (83 days since last) 2.2.027/10-25/11 29 days (101 days since last) 2.1.0

Re: Beam 2.4.0

About BEAM-3409, I did a review yesterday and it looks good to me. We are waiting for Thomas' feedback. Regards JB Le 1 mars 2018 à 09:38, à 09:38, Robert Bradshaw a écrit: >Looking at the burn-down list, we have 5 remaining issues. None of >these >are blockers, but all

Re: Beam 2.4.0

Looking at the burn-down list, we have 5 remaining issues. None of these are blockers, but all look like they're really close (reviewed, review comments were addressed, waiting for a final LGTM). Specifically: BEAM-3409 (teardown issues): Thomas Groh had some concerns, could you verify they have

Re: Beam 2.4.0

I tend to fall into the "release early, release often" camp in general, but for this one I'm particularly anxious to get the faster Python direct runner out in the hands of TFT/TFX users (and in particular have an eye on https://www.tensorflow.org/dev-summit/ which I think can be a healthy source

Re: Beam 2.4.0

Hi Gus, Thanks for the update, it makes sense. Regards JB On 03/01/2018 02:59 AM, Konstantinos Katsiapis wrote: > Hi Jean-Baptiste, > > I can speak from the perspective of tf.transform >  (TFT) in particular and TFX >

Re: Beam 2.4.0

Hi Jean-Baptiste, I can speak from the perspective of tf.transform (TFT) in particular and TFX libs in general, in case it is useful. TFX distributed computation has 2 "large" dependencies, namely

Re: Beam 2.4.0

By the way, if third party projects based on Beam (Google Dataflow, Talend DataStream, and others) need a release (including some features), it's better to clearly state this on the mailing list. At Apache Karaf, I have lot of projects based on it (OpenDaylight, OpenHAB, Websphere, ...). They

Re: Beam 2.4.0

A month is OK for me: I prefer that than longer (as it was before). The most important is a regular pace: we should avoid a 2.4.0 now and a 2.5.0 four months later. So, it means that 2.5.0 should happen roughly end of April. I will send a clear statement (as I already did weeks ago) around

Re: Beam 2.4.0

To be semver, 2.3.1 should be rollback safe with 2.3.0. Normally that is accomplished by cutting 2.3.1 release branch from 2.3.0 release branch and then have fixes cherrypicked. I think 6 weeks between minor version releases is not too fast. I think a month is a good target. We tend to have high

Re: Beam 2.4.0

That's the point of my e-mail indeed. I think it would make more sense for users. Regards JB On 02/28/2018 06:04 AM, Romain Manni-Bucau wrote: > Thinking out loud: why not trying a 2.3.1 with small fixes only and the 2.4 > after 6 weeks starting from the 2.3.0 real release date. > > Le 28

Re: Beam 2.4.0

Thinking out loud: why not trying a 2.3.1 with small fixes only and the 2.4 after 6 weeks starting from the 2.3.0 real release date. Le 28 févr. 2018 04:24, "Jean-Baptiste Onofré" a écrit : > OK, maybe I wasn't clear: for me the cycle is ~ 6 weeks once a release is > out, >

Re: Beam 2.4.0

OK, maybe I wasn't clear: for me the cycle is ~ 6 weeks once a release is out, not when it started. I don't remember about monthly release (it's too fast IMHO). Anyway, let me find the thread dealing with release pace and propose a clear statement. It's important for our users. Regards JB On

Re: Beam 2.4.0

It's been six weeks since you proposed beam 2.3.0. so assuming the same time scale for this release, that's 1.5 months between releases. Slightly faster than 2 months, but not by much. I do seem to remember that the original goal for beam was monthly releases though. Reuven On Tue, Feb 27,

Re: Beam 2.4.0

Hi Reuven, In a previous thread (about Beam project execution), I proposed a release every two months (as a best effort), I will find the e-mail. Beam 2.3.0 has been released "officially" on February 16th, so two week ago roughly. I would have expected 2.4.0 not before end of March. If we have

Re: Beam 2.4.0

Wasn't the original statement monthly releases? We've never realistically managed that, but Robert's proposed cut will be on a 6-week pace. On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste Onofré wrote: > Hi Robert, > > I'm just curious: it's pretty fast compared to the original

Re: Beam 2.4.0

Hi Robert, I'm just curious: it's pretty fast compared to the original plan of a release every two months. What's the reason to cut 2.4.0 now instead of end of March ? I will do the Jira triage and update today. Regards JB On 02/27/2018 09:21 PM, Robert Bradshaw wrote: > I'm planning on

Re: Beam 2.4.0

t makes direct runner - and therefore any >>> test - unusable at all (leaks+unexpected wait times + retry strategy >>> required+no real alternative even for dofn now dofntester is deprecated). >>> >>> Le 27 févr. 2018 21:45, "Chamikara Jayalath" <chamik...@

Re: Beam 2.4.0

quot;Chamikara Jayalath" <chamik...@google.com> a >> écrit : >> >>> Looks like previous URL was broken. Created https://s.apache.org/ >>> beam-2.4.0-burndown. >>> >>> - Cham >>> >>> On Tue, Feb 27, 2018 at 12:22 PM Robert Bradsha

Re: Beam 2.4.0

t;> required+no real alternative even for dofn now dofntester is deprecated). >> >> Le 27 févr. 2018 21:45, "Chamikara Jayalath" <chamik...@google.com> a >> écrit : >> >>> Looks like previous URL was broken. Created >>> https://s.apache.org/be

Re: Beam 2.4.0

: > >> Looks like previous URL was broken. Created >> https://s.apache.org/beam-2.4.0-burndown. >> >> - Cham >> >> On Tue, Feb 27, 2018 at 12:22 PM Robert Bradshaw <rober...@google.com> >> wrote: >> >>> I'm planning on cutting the 2.

Re: Beam 2.4.0

.com> a écrit : > Looks like previous URL was broken. Created https://s.apache.org/ > beam-2.4.0-burndown. > > - Cham > > On Tue, Feb 27, 2018 at 12:22 PM Robert Bradshaw <rober...@google.com> > wrote: > >> I'm planning on cutting the 2.4.0 release branch soon (tomo

Re: Beam 2.4.0

Looks like previous URL was broken. Created https://s.apache.org/beam-2.4.0-burndown. - Cham On Tue, Feb 27, 2018 at 12:22 PM Robert Bradshaw <rober...@google.com> wrote: > I'm planning on cutting the 2.4.0 release branch soon (tomorrow?). I see 13 > open issues on JIRA [1], n

Beam 2.4.0

I'm planning on cutting the 2.4.0 release branch soon (tomorrow?). I see 13 open issues on JIRA [1], none of which are labeled as blockers. If there are any that cannot be bumped to the next release, let me know soon. - Robert [1]

Re: Beam 2.4.0

done Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book

Re: Beam 2.4.0

Romain - it looks like these JIRA tickets are on on the 2.4.0 burndown. Can you set their Fix Version field to make sure they are tracked and triaged? On Tue, Feb 20, 2018 at 10:22 PM, Reuven Lax wrote: > I think it's fair to request that the reviewers of these PRs help with

Re: Beam 2.4.0

*are _not_ on the burndown :-) On Wed, Feb 21, 2018 at 9:31 AM, Kenneth Knowles wrote: > Romain - it looks like these JIRA tickets are on on the 2.4.0 burndown. > Can you set their Fix Version field to make sure they are tracked and > triaged? > > > On Tue, Feb 20, 2018 at

Re: Beam 2.4.0

I think it's fair to request that the reviewers of these PRs help with your effort to get them merged before the 2.4.0 cut. Existing comments on the PR imply that reviewers think the approaches are reasonable. Assuming that there's not too much work left to be done to address comments, there's a

Re: Beam 2.4.0

Ok In terms of what I'd like included, here is the list: 1. https://github.com/apache/beam/pull/4412 (important to prevent regressions) 2. https://github.com/apache/beam/pull/4674 (can need some more work but can break some api if we do, so current state is a functional trade off). On a more

Re: Beam 2.4.0

+1, this is keeping with an every-six weeks cadence. Romain, you can always target Jiras to this release, and then the release manager can decide on a case-by-case basis whether to make sure the fix is included. On Tue, Feb 20, 2018 at 2:30 PM, Robert Bradshaw wrote: >

Re: Beam 2.4.0

+1 on having release trains scheduled. Romain: Do you have a list of PRs that could benefit from increased focus if they want to make it on the upcoming train? On Tue, Feb 20, 2018 at 3:30 PM Ahmet Altay wrote: > +1 for having regular release cycles. Finalizing a release

Re: Beam 2.4.0

+1 for having regular release cycles. Finalizing a release takes time in the order of a few weeks and starting a new release soon after the previous one is a reliable way for having releases every 6 weeks. On Tue, Feb 20, 2018 at 2:30 PM, Robert Bradshaw wrote: > Yep. I am

Re: Beam 2.4.0

Yep. I am starting the "Let's do a 2.4.0 release" thread almost exactly 6 weeks after JB first started the 2.3.0 release thread. On Tue, Feb 20, 2018 at 2:20 PM, Charles Chen wrote: > I would like to +1 the faster release cycle process JB and Robert have been > advocating and

Re: Beam 2.4.0

I would like to +1 the faster release cycle process JB and Robert have been advocating and implementing, and thank JB for releasing 2.3.0 smoothly. When we block for specific features and increase the time between releases, we increase the urgency for PR authors to push for their change to go into

Re: Beam 2.4.0

Kind of agree but rythm was supposed to be 6 weeks IIRC, 2.3 is just out so 1 week is a bit fast IMHO. Le 20 févr. 2018 23:13, "Robert Bradshaw" a écrit : > One of the main shifts that I think helped this release was explicitly > not being feature driven, rather releasing

Re: Beam 2.4.0

One of the main shifts that I think helped this release was explicitly not being feature driven, rather releasing what's already in the branch. That doesn't mean it's not a good call to action to try and get long-pending PRs or similar wrapped up. On Tue, Feb 20, 2018 at 2:10 PM, Romain

Re: Beam 2.4.0

There are a lot of long pending PR, would be good to merge them before 2.4. Some are bringing tests for the 2.3 release which can be critical to include. Maybe we should list the pr and jira we want it before picking a date? Le 20 févr. 2018 22:02, "Konstantinos Katsiapis"

Re: Beam 2.4.0

+1 since tf.transform 0.6 depends on Beam 2.4 and Tensorflow 1.6 (and the latter already has an RC out, so we will likely be blocked on Beam). On Tue, Feb 20, 2018 at 12:50 PM, Robert Bradshaw wrote: > Now that Beam 2.3.0 went out

Beam 2.4.0

Now that Beam 2.3.0 went out (and in record time, kudos to all that made this happen!) It'd be great to keep the ball rolling for a similarly well-executed 2.4. A lot has gone in [1] since we made the 2.3 cut, and to keep our cadence up I would propose a time-based cut date early next week (say