Hi guys, tried to reapply the waitUntilFinish fix - without breaking the compilation this time ;) - in https://github.com/apache/beam/pull/4790, anyone able to help to review and move that forward?
Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book <https://www.packtpub.com/application-development/java-ee-8-high-performance> 2018-03-02 9:28 GMT+01:00 Robert Bradshaw <rober...@google.com>: > On Fri, Mar 2, 2018 at 12:01 AM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > >> Hi Ismaël, >> >> that's a good idea to show history. >> >> For me, the vote duration doesn't matter as we are in the release process >> already. >> > > A more relevant duration to track would probably be cut to final release. > This both measures our investment in the release process, as well as how > behind head is when we finally get the release out. > > >> The gap between two releases is more significant. > > > +1, this is what users see. (To clarify terminology, the "time between > release" is the time between actually releasing x.y and x.y+1 that is most > visible to end users, regardless of intermediate process like cutting and > voting that we have.) Of course this gets thrown off if our > time-to-prepare-a-release becomes a significant fraction of the desired > time-between-releases. > > And clearly with an average of >> 80 days (~ 3 months) it's two long. The idea is to reduce this clearly. I >> propose two months previously (including the vote period), so meaning 6 >> weeks >> between releases. >> > > It seems there have been proposals for monthly, every 6 weeks > (sesquimonthly?), and bimonthly. > > >> Regarding the time we take for the PR review and merge, I think it's a >> fair time >> to give us time to include new improvements and features, but also to fix >> bugs. >> > > Concrete deadlines can provide good motivation to get around to doing > reviews, cleaning up PRs, fixing bugs, etc. that you've been meaning to do > but for whatever reason keep putting off. So I think it's still good > practice to have some lead time that a release is coming for a chance for > folks to get stuff in, while still being clear that we're not holding > things back for new features and if you don't make the cut another one is > close behind. > > Regards >> JB >> >> On 03/01/2018 06:17 PM, Ismaël Mejía wrote: >> > The average time just in the vote process for Beam since we are out of >> the >> > incubator is 17.5 days with an average of 75 days between versions. >> > >> > Version Vote Period No. days >> > 2.3.0 30/01-16/02 17 days (83 days since last) >> > 2.2.0 27/10-25/11 29 days (101 days since last) >> > 2.1.0 11/07-16/08 36 days (92 days since last) >> > 2.0.0 08/05-16/05 8 days (62 days since last) >> > 0.6.0 10/03-15/03 5 days (37 days since last) >> > 0.5.0 27/01-06/02 10 days >> > >> > I think we should have these numbers into account to refine the >> distance between >> > releases. If we want to follow strict time-based releases, what we can >> probably >> > refine is how we cut the release so we try to reduce release overlaps >> and avoid >> > rushing unnecessarily. >> > >> > Maybe we should follow the proposed 6 weeks for the next release like >> this: >> > >> > - 4 weeks let’s say just after succesful vote and then cut the release >> branch. >> > - 1 week to burn the blocker list (good to have ones that don’t make >> will be >> > moved to the next release). >> > - 1 week for the vote + RCs (in case the vote takes longer at least the >> overlap >> > between vote + next dev cycle will be smaller). >> > >> > If we do this for the next cycle we will have a 6 week ‘dev’ period and >> then we >> > will have optimistically an average of 2 weeks of ‘releasing’ and 6 >> weeks ‘dev’ >> > cycles. >> > >> > On Thu, Mar 1, 2018 at 6:46 AM, Jean-Baptiste Onofré <j...@nanthrax.net> >> wrote: >> >> About BEAM-3409, I did a review yesterday and it looks good to me. We >> are >> >> waiting for Thomas' feedback. >> >> >> >> Regards >> >> JB >> >> Le 1 mars 2018, à 09:38, Robert Bradshaw <rober...@google.com> a >> écrit: >> >>> >> >>> Looking at the burn-down list, we have 5 remaining issues. None of >> these >> >>> are blockers, but all look like they're really close (reviewed, review >> >>> comments were addressed, waiting for a final LGTM). Specifically: >> >>> >> >>> BEAM-3409 (teardown issues): Thomas Groh had some concerns, could you >> >>> verify they have all been addressed? >> >>> BEAM-3479 (DoFn classloader regression test): Kenn Knowles had minor >> >>> comments, looks like they were addressed, could you confirm? >> >>> BEAM-3735 (Missing gaming release archetypes): Lukasz Cwik had minor >> >>> comments, looks like they were addressed, could you confirm? >> >>> BEAM-3611 (KafkaIO.java splitting): Looks like this was resolved. >> >>> BEAM-3762 (unlimited JCE for Dataflow Worker): LGTM pending (currently >> >>> running) tests passing. >> >>> >> >>> Let's see how many of these we can get in by, say, noon PST tomorrow. >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> On Wed, Feb 28, 2018 at 9:26 PM Robert Bradshaw < rober...@google.com >> > >> >>> wrote: >> >>>> >> >>>> I tend to fall into the "release early, release often" camp in >> general, >> >>>> but for this one I'm particularly anxious to get the faster Python >> direct >> >>>> runner out in the hands of TFT/TFX users (and in particular have an >> eye on >> >>>> https://www.tensorflow.org/dev-summit/ which I think can be a >> healthy source >> >>>> of Beam users). >> >>>> >> >>>> >> >>>> On Wed, Feb 28, 2018 at 7:01 PM Jean-Baptiste Onofré < >> j...@nanthrax.net> >> >>>> wrote: >> >>>>> >> >>>>> Hi Gus, >> >>>>> >> >>>>> Thanks for the update, it makes sense. >> >>>>> >> >>>>> Regards >> >>>>> JB >> >>>>> >> >>>>> On 03/01/2018 02:59 AM, Konstantinos Katsiapis wrote: >> >>>>>> Hi Jean-Baptiste, >> >>>>>> >> >>>>>> I can speak from the perspective of tf.transform >> >>>>>> < https://github.com/tensorflow/transform> (TFT) in particular >> and TFX >> >>>>>> < https://research.google.com/pubs/pub46484.html> libs in >> general, in >> >>>>>> case it is >> >>>>>> useful. >> >>>>>> >> >>>>>> TFX distributed computation has 2 "large" dependencies, namely >> >>>>>> TensorFlow and >> >>>>>> Apache Beam, each on their own release schedule. >> >>>>>> As such, releasing of new TFX functionality often (but not always) >> >>>>>> depends on >> >>>>>> (and is blocked by) releases of *both* TensorFlow *and* Apache >> Beam. >> >>>>>> >> >>>>>> Synchronizing releases across such large projects and >> organizations is >> >>>>>> likely >> >>>>>> hard, so from our perspective having *frequent* releases of >> Tensorflow >> >>>>>> or Apache >> >>>>>> Beam (and better yet both) decreases the time for which we are >> blocked >> >>>>>> on >> >>>>>> releasing our features. >> >>>>>> >> >>>>>> In light of this, I would vote for more frequent releases in >> general, >> >>>>>> and for a >> >>>>>> Beam 2.4 release soon in particular (as TFT 0.6 depends on it). >> >>>>>> >> >>>>>> Thanks, >> >>>>>> Gus >> >>>>>> >> >>>>>> On Tue, Feb 27, 2018 at 10:29 PM, Jean-Baptiste Onofré < >> >>>>>> j...@nanthrax.net >> >>>>>> <mailto: j...@nanthrax.net>> wrote: >> >>>>>> >> >>>>>> By the way, if third party projects based on Beam (Google >> >>>>>> Dataflow, Talend >> >>>>>> DataStream, and others) need a release (including some >> features), >> >>>>>> it's better to >> >>>>>> clearly state this on the mailing list. >> >>>>>> >> >>>>>> At Apache Karaf, I have lot of projects based on it >> (OpenDaylight, >> >>>>>> OpenHAB, >> >>>>>> Websphere, ...). They just ask for the release schedule and >> they >> >>>>>> align with >> >>>>>> these release. As a best effort, I'm always trying to move fast >> >>>>>> when a release >> >>>>>> is asked. >> >>>>>> >> >>>>>> So, if 2.4.0 is required by third party, no problem to "ask >> for a >> >>>>>> release". >> >>>>>> >> >>>>>> Regards >> >>>>>> JB >> >>>>>> >> >>>>>> On 02/28/2018 04:17 AM, Reuven Lax wrote: >> >>>>>> > It's been six weeks since you proposed beam 2.3.0. so >> assuming >> >>>>>> the same time >> >>>>>> > scale for this release, that's 1.5 months between releases. >> >>>>>> Slightly faster than >> >>>>>> > 2 months, but not by much. >> >>>>>> > >> >>>>>> > I do seem to remember that the original goal for beam was >> >>>>>> monthly releases though. >> >>>>>> > >> >>>>>> > Reuven >> >>>>>> > >> >>>>>> > On Tue, Feb 27, 2018, 9:12 PM Jean-Baptiste Onofré < >> >>>>>> j...@nanthrax.net <mailto: j...@nanthrax.net> >> >>>>>> > <mailto: j...@nanthrax.net <mailto: j...@nanthrax.net>>> wrote: >> >>>>>> > >> >>>>>> > Hi Reuven, >> >>>>>> > >> >>>>>> > In a previous thread (about Beam project execution), I >> >>>>>> proposed a release every >> >>>>>> > two months (as a best effort), I will find the e-mail. >> >>>>>> > >> >>>>>> > Beam 2.3.0 has been released "officially" on February >> 16th, >> >>>>>> so two week ago >> >>>>>> > roughly. I would have expected 2.4.0 not before end of >> >>>>>> March. >> >>>>>> > >> >>>>>> > If we have issue we want to fix fast, then 2.3.1 is >> good. If >> >>>>>> it's a new release >> >>>>>> > in the pace, it's pretty fast and might "confuse" our >> users. >> >>>>>> > >> >>>>>> > That's why I'm curious ;) >> >>>>>> > >> >>>>>> > Regards >> >>>>>> > JB >> >>>>>> > >> >>>>>> > On 02/28/2018 03:50 AM, Reuven Lax wrote: >> >>>>>> > > Wasn't the original statement monthly releases? We've >> >>>>>> never realistically >> >>>>>> > > managed that, but Robert's proposed cut will be on a >> >>>>>> 6-week pace. >> >>>>>> > > >> >>>>>> > > On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste Onofré < >> >>>>>> j...@nanthrax.net <mailto: j...@nanthrax.net> >> >>>>>> > <mailto: j...@nanthrax.net <mailto: j...@nanthrax.net>> >> >>>>>> > > <mailto: j...@nanthrax.net <mailto: j...@nanthrax.net> >> >>>>>> <mailto: j...@nanthrax.net >> >>>>>> <mailto: j...@nanthrax.net>>>> wrote: >> >>>>>> > > >> >>>>>> > > Hi Robert, >> >>>>>> > > >> >>>>>> > > I'm just curious: it's pretty fast compared to the >> >>>>>> original plan of a >> >>>>>> > release >> >>>>>> > > every two months. What's the reason to cut 2.4.0 >> now >> >>>>>> instead of end of >> >>>>>> > March ? >> >>>>>> > > >> >>>>>> > > I will do the Jira triage and update today. >> >>>>>> > > >> >>>>>> > > Regards >> >>>>>> > > JB >> >>>>>> > > >> >>>>>> > > On 02/27/2018 09:21 PM, Robert Bradshaw wrote: >> >>>>>> > > > I'm planning on cutting the 2.4.0 release branch >> >>>>>> soon (tomorrow?). I >> >>>>>> > see 13 >> >>>>>> > > > open issues on JIRA [1], none of which are >> labeled >> >>>>>> as blockers. If there >> >>>>>> > > > are any that cannot be bumped to the next >> release, >> >>>>>> let me know soon. >> >>>>>> > > > >> >>>>>> > > > - Robert >> >>>>>> > > > >> >>>>>> > > > >> >>>>>> > > > [1] >> >>>>>> > > > >> >>>>>> > > >> >>>>>> > >> >>>>>> https://issues.apache.org/jira/browse/BEAM-3749?jql= >> project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20% >> 22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0 >> >>>>>> < >> >>>>>> https://issues.apache.org/jira/browse/BEAM-3749?jql= >> project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20% >> 22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0> >> >>>>>> > > > >> >>>>>> > > >> >>>>>> > > -- >> >>>>>> > > Jean-Baptiste Onofré >> >>>>>> > > jbono...@apache.org <mailto: jbono...@apache.org> >> >>>>>> <mailto: jbono...@apache.org >> >>>>>> <mailto: jbono...@apache.org>> >> >>>>>> > <mailto: jbono...@apache.org <mailto: >> jbono...@apache.org> >> >>>>>> <mailto: jbono...@apache.org <mailto: jbono...@apache.org>>> >> >>>>>> > > http://blog.nanthrax.net >> >>>>>> > > Talend - http://www.talend.com >> >>>>>> > > >> >>>>>> > >> >>>>>> > -- >> >>>>>> > Jean-Baptiste Onofré >> >>>>>> > jbono...@apache.org <mailto: jbono...@apache.org> >> >>>>>> <mailto: jbono...@apache.org <mailto: jbono...@apache.org>> >> >>>>>> > http://blog.nanthrax.net >> >>>>>> > Talend - http://www.talend.com >> >>>>>> > >> >>>>>> >> >>>>>> -- >> >>>>>> Jean-Baptiste Onofré >> >>>>>> jbono...@apache.org <mailto: jbono...@apache.org> >> >>>>>> http://blog.nanthrax.net >> >>>>>> Talend - http://www.talend.com >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> -- >> >>>>>> Gus Katsiapis | Software Engineer | katsia...@google.com >> >>>>>> <mailto: katsia...@google.com> | 650-918-7487 >> >>>>> >> >>>>> -- >> >>>>> Jean-Baptiste Onofré >> >>>>> jbono...@apache.org >> >>>>> http://blog.nanthrax.net >> >>>>> Talend - http://www.talend.com >> >> -- >> Jean-Baptiste Onofré >> jbono...@apache.org >> http://blog.nanthrax.net >> Talend - http://www.talend.com >> >
