Re: Apache Beam 2.4.0 release process retrospective and automation possibilities

[Romain Manni-Bucau]

2018-03-23 9:52 GMT+01:00 Robert Bradshaw :

> To put this in context, this was a brain dump of some of the things I
> encountered while doing the release. Were I to do a release again, it would
> be a lot easier (though still not ideal).
>
> At the high level, rather than focusing on steps, I think it's more
> interesting to focus on why we need a human to do the release. IMHO, the
> role of a human is
>
> 1) Choose a commit. Any commit in master should do, if we had proper test
> and code hygiene.
> 2) Sign the release artifacts. (Frankly, I would be happy with a robot
> signing everything but the tag in github, if the keys could be properly
> maintained.)
> 3) Manage the email thread.
>
> To this end, I would like a process where one would propose a release
> candidate via a lightweight, fast tool. and jenkins (or some other system)
> would recognize the release branch and test, generate artifacts, push
> artifacts, and test against those artifacts (including, ideally, nexus and
> svn staging). One should be able to easily run this locally, but that
> shouldn't be needed. If a human needs to sign, there could be a script for
> one to download artifacts, sign them, and push the signatures. On success,
> an email body (with all the links and details) would be created, to be sent
> out by a human.
>
> More responses inline below (and thanks for the feedback!).
>
>
> On Thu, Mar 22, 2018 at 11:10 PM Romain Manni-Bucau 
> wrote:
>
>> Hi
>>
>> Le 23 mars 2018 04:29, "Alan Myrvold"  a écrit :
>>
>> Robert explained his experience with the release process
>>  as the release
>> engineer for 2.4.0, and we discussed the prototype shell script for
>> checking release progress in pull/4896
>> .
>>
>> I'd like to help automate the release process, initially just checking
>> that all steps look ok, then automating all feasible steps, with the goal
>> of reducing the effort of the release engineer per release to less than 1
>> hour for creating the first RC.
>>
>> Overall, it is a large, scattered process. For someone who has done this
>> many times (like jb@ in the previous release), it is likely easy. Robert
>> was familiar with pypi, making that part easy for him. He was not as
>> familiar with the java release artifacts or nexus, making that more of a
>> challenge.
>>
>>- Several steps are not reversible, so it isn't a restartable process
>>if there are errors.
>>
>> Can we ensure they are done last? Maven introduced the deployAtEnd
>> feature so i guess it can be a model used here.
>>
>
> In retrospect, aside from pushing stuff, things weren't strictly
> irreversible. I got to be good friends with git clean, git checkout, and
> git reset. That last one, however, always feels wrong to use (and is
> error-prone).
>
> I also don't think I'll ever get used to the build system (mvn or gradle)
> editing code and creating commits, but perhaps that's needed with the way
> -SNAPSHOTS are special (though having to edit every file seems overkill).
>

it is fully automated and hidden behind mvn release:prepare normally
(gradle as the equivalent but its design is a bit different so it can be a
bit more natural for some people), did you do it manually?


>
>>- Several steps have high latency; it may take 30min of work before
>>prompting for a password.
>>
>> What prevents to put the pwd in settings.xml or use an agent - to not be
>> prompted?
>>
>
> Foreknowledge :). I did end up using an agent. (Personally, I have some
> qualms putting my gpg password in a plaintext xml file). There were also
> gitbox prompts in some of the steps, and of course svn commit (though the
> svn steps weren't high latency). can't remember if there were others.
>

Can it be worth a word in the doc? Side note: on linux you dont have to put
it in clear.


>
>>- Problems with both his laptop and desktop; GCS wasn't working well
>>on the laptop preventing running tests. Maven extremely slow on his
>>desktop, but he discovered a workaround in his configuration. Would have
>>been nice to use jenkins for most steps instead of relying on
>>laptop/desktop configurations.
>>
>> As JB mentioned, these were environmental issues. But it wasn't clear at
> the time (being new with the release process) and could have been elevated
> had I not had to do so many manual steps (including running the tests). I
> happened to get particularly unlucky with timing with one of them too.
>
>>
>>-
>>- Many of the steps ran the same tests over and over. Sometimes tests
>>were flaky, so needed to restart a long process due to a test that had
>>passed earlier now flaking.
>>
>> Wonder if remote tests shouldnt be mocked *during* a release to avoid
>> that no luck effect.
>>
>
> We should be able to release from a known good commit, as verified by
> jenkins, and never run 

Re: Apache Beam 2.4.0 release process retrospective and automation possibilities

[Robert Bradshaw]

To put this in context, this was a brain dump of some of the things I
encountered while doing the release. Were I to do a release again, it would
be a lot easier (though still not ideal).

At the high level, rather than focusing on steps, I think it's more
interesting to focus on why we need a human to do the release. IMHO, the
role of a human is

1) Choose a commit. Any commit in master should do, if we had proper test
and code hygiene.
2) Sign the release artifacts. (Frankly, I would be happy with a robot
signing everything but the tag in github, if the keys could be properly
maintained.)
3) Manage the email thread.

To this end, I would like a process where one would propose a release
candidate via a lightweight, fast tool. and jenkins (or some other system)
would recognize the release branch and test, generate artifacts, push
artifacts, and test against those artifacts (including, ideally, nexus and
svn staging). One should be able to easily run this locally, but that
shouldn't be needed. If a human needs to sign, there could be a script for
one to download artifacts, sign them, and push the signatures. On success,
an email body (with all the links and details) would be created, to be sent
out by a human.

More responses inline below (and thanks for the feedback!).


On Thu, Mar 22, 2018 at 11:10 PM Romain Manni-Bucau 
wrote:

> Hi
>
> Le 23 mars 2018 04:29, "Alan Myrvold"  a écrit :
>
> Robert explained his experience with the release process
>  as the release
> engineer for 2.4.0, and we discussed the prototype shell script for
> checking release progress in pull/4896
> .
>
> I'd like to help automate the release process, initially just checking
> that all steps look ok, then automating all feasible steps, with the goal
> of reducing the effort of the release engineer per release to less than 1
> hour for creating the first RC.
>
> Overall, it is a large, scattered process. For someone who has done this
> many times (like jb@ in the previous release), it is likely easy. Robert
> was familiar with pypi, making that part easy for him. He was not as
> familiar with the java release artifacts or nexus, making that more of a
> challenge.
>
>- Several steps are not reversible, so it isn't a restartable process
>if there are errors.
>
> Can we ensure they are done last? Maven introduced the deployAtEnd feature
> so i guess it can be a model used here.
>

In retrospect, aside from pushing stuff, things weren't strictly
irreversible. I got to be good friends with git clean, git checkout, and
git reset. That last one, however, always feels wrong to use (and is
error-prone).

I also don't think I'll ever get used to the build system (mvn or gradle)
editing code and creating commits, but perhaps that's needed with the way
-SNAPSHOTS are special (though having to edit every file seems overkill).

>
>- Several steps have high latency; it may take 30min of work before
>prompting for a password.
>
> What prevents to put the pwd in settings.xml or use an agent - to not be
> prompted?
>

Foreknowledge :). I did end up using an agent. (Personally, I have some
qualms putting my gpg password in a plaintext xml file). There were also
gitbox prompts in some of the steps, and of course svn commit (though the
svn steps weren't high latency). can't remember if there were others.

>
>- Problems with both his laptop and desktop; GCS wasn't working well
>on the laptop preventing running tests. Maven extremely slow on his
>desktop, but he discovered a workaround in his configuration. Would have
>been nice to use jenkins for most steps instead of relying on
>laptop/desktop configurations.
>
> As JB mentioned, these were environmental issues. But it wasn't clear at
the time (being new with the release process) and could have been elevated
had I not had to do so many manual steps (including running the tests). I
happened to get particularly unlucky with timing with one of them too.

>
>-
>- Many of the steps ran the same tests over and over. Sometimes tests
>were flaky, so needed to restart a long process due to a test that had
>passed earlier now flaking.
>
> Wonder if remote tests shouldnt be mocked *during* a release to avoid that
> no luck effect.
>

We should be able to release from a known good commit, as verified by
jenkins, and never run tests again.


>
>- Robert was new to Nexus, so setting up permissions and navigating
>the UI was confusing.
>
> When it happens dont hesitate to ping here.
>
>
>-
>- Needed to rebuild the google cloud dataflow containers to get
>dataflow working with the RC, and that ended up being a painful process.
>The github sdk/python/container is part of the portability effort and
>should help eliminate googlers needing to do steps like this with each
>release because that container 

Re: Apache Beam 2.4.0 release process retrospective and automation possibilities

[Jean-Baptiste Onofré]

Hi Alan,

some feedback inline:

>   * Several steps have high latency; it may take 30min of work before 
> prompting
> for a password.

Using GPG agent or  in .m2/settings.xml works fine. It's what
I'm doing in all Apache releases I'm doing.

>   * Problems with both his laptop and desktop; GCS wasn't working well on the
> laptop preventing running tests. Maven extremely slow on his desktop, but 
> he
> discovered a workaround in his configuration. Would have been nice to use
> jenkins for most steps instead of relying on laptop/desktop 
> configurations.

Not sure there, the release has to be done by the release manager IMHO. So,
that's an environment issue, not a release issue.

>   * Many of the steps ran the same tests over and over. Sometimes tests were
> flaky, so needed to restart a long process due to a test that had passed
> earlier now flaking.

Good point, definitely improvements on the tests to do (maybe better split
between utests and itests).

>   * Robert was new to Nexus, so setting up permissions and navigating the UI 
> was
> confusing.

Not a release concern IMHO, Nexus is straight forward for release with staging
repo. I think it's well explained in the release guide.

>   * Needed to rebuild the google cloud dataflow containers to get dataflow
> working with the RC, and that ended up being a painful process. The github
> sdk/python/container is part of the portability effort and should help
> eliminate googlers needing to do steps like this with each release because
> that container can be built externally.

+1

>   * Automated release notes were not seen as valuable due to limitations in 
> any
> automated documentation.

I guess you mean Release Notes available on Jira, so I agree, it's already
easily generated.

>   * Missed a step of changing the java/python version numbers but was able to
> fix that.

Changing version numbers where ? Release plugin (in the prepare goal) already
change the versions in the POMs (like a mvn versions:set). I guess you are
talking about some other files ? Maybe those files could use the project.version
from the pom ?

>   * Some copy/paste errors when creating the voting emails.>   * Many steps 
> are not possible for non-committers.

And it's normal per Apache rule.

Regards
JB
-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Apache Beam 2.4.0 release process retrospective and automation possibilities

[Romain Manni-Bucau]

Hi

Le 23 mars 2018 04:29, "Alan Myrvold"  a écrit :

Robert explained his experience with the release process
 as the release engineer
for 2.4.0, and we discussed the prototype shell script for checking release
progress in pull/4896 .

I'd like to help automate the release process, initially just checking that
all steps look ok, then automating all feasible steps, with the goal of
reducing the effort of the release engineer per release to less than 1 hour
for creating the first RC.

Overall, it is a large, scattered process. For someone who has done this
many times (like jb@ in the previous release), it is likely easy. Robert
was familiar with pypi, making that part easy for him. He was not as
familiar with the java release artifacts or nexus, making that more of a
challenge.

   - Several steps are not reversible, so it isn't a restartable process if
   there are errors.


Can we ensure they are done last? Maven introduced the deployAtEnd feature
so i guess it can be a model used here.


   -
   - Several steps have high latency; it may take 30min of work before
   prompting for a password.


What prevents to put the pwd in settings.xml or use an agent - to not be
prompted?


   -
   - Problems with both his laptop and desktop; GCS wasn't working well on
   the laptop preventing running tests. Maven extremely slow on his desktop,
   but he discovered a workaround in his configuration. Would have been nice
   to use jenkins for most steps instead of relying on laptop/desktop
   configurations.
   - Many of the steps ran the same tests over and over. Sometimes tests
   were flaky, so needed to restart a long process due to a test that had
   passed earlier now flaking.

Wonder if remote tests shouldnt be mocked *during* a release to avoid that
no luck effect.


   -
   - Robert was new to Nexus, so setting up permissions and navigating the
   UI was confusing.

When it happens dont hesitate to ping here.


   -
   - Needed to rebuild the google cloud dataflow containers to get dataflow
   working with the RC, and that ended up being a painful process. The github
   sdk/python/container is part of the portability effort and should help
   eliminate googlers needing to do steps like this with each release because
   that container can be built externally.

Is there a way to see this process or is it "closed"?


   -
   - Automated release notes were not seen as valuable due to limitations
   in any automated documentation.


If needed i have a script to do it from jira, i filter issue by the label
"changelog". RM must review tickets before the release with that.


   - Missed a step of changing the java/python version numbers but was able
   to fix that.

Is it doable through maven/gradle filtering?


   - Some copy/paste errors when creating the voting emails.


At tomee we had a cli to do it if interested.


   - Many steps are not possible for non-committers.

Sadly intended/expected but you cannhelp on jira review, snapshot
validation before the release.


   - The prototype shell script was seen as helpful, especially since it
   can be restarted. Some concerns over the maintainability of such a large
   shell script.

Move to groovy? Or is it just a size issue?

The steps that should not be automated, and need human involvement:

   1. All emails (propose release, ask for votes)
   2. Picking the commit to start the release
   3. Signing artifacts

Most everything else should be possible to automate, although
non-committers do not have access to logging into jira, nexus, or the
jenkins ui, making some of this tricky to automate for non-contributors.
Also not clear to us how nexus picks he sequential artifact suffix (1031)?


It is a sequence - as db sequence, dont recall if it is per user or repo
(repo from memory but not 100% sure). If that is to automate close and
mailing maybe check sonatype plugin which replace deploy one - ensure to
deactivate auto release - or just nexus api which returns this value. A
maven (gradle) extension should be writable too.


Next steps for me are to enhance the release-checking script, automate
feasible actions, and pair with the next release engineer to make this
smoother, especially if they are at google, but even if they are not.


+1, anything should be bound to release:perform except the vote process and
the postvote tasks which can be automated (dist update, site update,
release staging, )


Alan

Re: [ANNOUNCE] Apache Beam 2.4.0 released

[Romain Manni-Bucau]

congrats guys


Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> |  Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>

2018-03-22 9:50 GMT+01:00 Etienne Chauchot <echauc...@apache.org>:

> Great !
> Le jeudi 22 mars 2018 à 08:24 +, Robert Bradshaw a écrit :
>
> We are pleased to announce the release of Apache Beam 2.4.0. Thanks goes to
> the many people who made this possible.
>
> Apache Beam is an open source unified programming model to define and
> execute data processing pipelines, including ETL, batch and stream
> (continuous) processing. See https://beam.apache.org
>
> You can download the release here:
>
>  https://beam.apache.org/get-started/downloads/
>
> As well as many bugfixes, some notable changes in this release are:
> - A new Python Direct runner, up to 15x faster than the old one.
> - Kinesis support for reading and writing in Java
> - Several refactoring to enable portability (Go/Python on Flink/Spark)
>
> Full release notes can be found at
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12342682=12319527
>
> Enjoy!
>
>

Re: [ANNOUNCE] Apache Beam 2.4.0 released

[Etienne Chauchot]

Great !
Le jeudi 22 mars 2018 à 08:24 +, Robert Bradshaw a écrit :
> We are pleased to announce the release of Apache Beam 2.4.0. Thanks goes to
> the many people who made this possible.
> 
> Apache Beam is an open source unified programming model to define and
> execute data processing pipelines, including ETL, batch and stream
> (continuous) processing. See https://beam.apache.org
> 
> You can download the release here:
> 
>  https://beam.apache.org/get-started/downloads/
> 
> As well as many bugfixes, some notable changes in this release are:
> - A new Python Direct runner, up to 15x faster than the old one.
> - Kinesis support for reading and writing in Java
> - Several refactoring to enable portability (Go/Python on Flink/Spark)
> 
> Full release notes can be found at
> 
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12342682=12319527
> 
> Enjoy!

[ANNOUNCE] Apache Beam 2.4.0 released

[Robert Bradshaw]

We are pleased to announce the release of Apache Beam 2.4.0. Thanks goes to
the many people who made this possible.

Apache Beam is an open source unified programming model to define and
execute data processing pipelines, including ETL, batch and stream
(continuous) processing. See https://beam.apache.org

You can download the release here:

 https://beam.apache.org/get-started/downloads/

As well as many bugfixes, some notable changes in this release are:
- A new Python Direct runner, up to 15x faster than the old one.
- Kinesis support for reading and writing in Java
- Several refactoring to enable portability (Go/Python on Flink/Spark)

Full release notes can be found at

https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12342682=12319527

Enjoy!

Re: Beam 2.4.0

[Romain Manni-Bucau]

Done, thanks.


Romain Manni-Bucau
@rmannibucau  |  Blog
 | Old Blog
 | Github  |
LinkedIn  | Book


2018-03-15 2:26 GMT+01:00 Jean-Baptiste Onofré :

> Fully agree.
>
> If nobody jumps on it, I will tackle that tomorrow.
>
> Regards
> JB
> Le 14 mars 2018, à 18:03, Reuven Lax  a écrit:
>>
>> Can you add a description and assign a reviewer (using R:). I think this
>> was basically ready to merge before modulo breaking compilation.
>>
>>
>> On Wed, Mar 14, 2018 at 10:01 AM Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> up, know it missed the 2.4 but can https://github.com/apache/
>>> beam/pull/4790 have some love? it really makes beam pretty unusable
>>> with direct runner,
>>> I start to have "// workaround for BEAM-3409" everywhere in my codebase
>>> which is quite bothering after 3 months :(
>>>
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau  |  Blog
>>>  | Old Blog
>>>  | Github
>>>  | LinkedIn
>>>  | Book
>>> 
>>>
>>> 2018-03-06 14:44 GMT+01:00 Romain Manni-Bucau :
>>>
 Hi guys,

 tried to reapply the waitUntilFinish fix - without breaking the
 compilation this time ;) - in https://github.com/apache/beam/pull/4790,
 anyone able to help to review and move that forward?


 Romain Manni-Bucau
 @rmannibucau  |  Blog
  | Old Blog
  | Github
  | LinkedIn
  | Book
 

 2018-03-02 9:28 GMT+01:00 Robert Bradshaw :

> On Fri, Mar 2, 2018 at 12:01 AM Jean-Baptiste Onofré 
> wrote:
>
>> Hi Ismaël,
>>
>> that's a good idea to show history.
>>
>> For me, the vote duration doesn't matter as we are in the release
>> process already.
>>
>
> A more relevant duration to track would probably be cut to final
> release. This both measures our investment in the release process, as well
> as how behind head is when we finally get the release out.
>
>
>> The gap between two releases is more significant.
>
>
> +1, this is what users see. (To clarify terminology, the "time between
> release" is the time between actually releasing x.y and x.y+1 that is most
> visible to end users, regardless of intermediate process like cutting and
> voting that we have.) Of course this gets thrown off if our
> time-to-prepare-a-release becomes a significant fraction of the desired
> time-between-releases.
>
> And clearly with an average of
>> 80 days (~ 3 months) it's two long. The idea is to reduce this
>> clearly. I
>> propose two months previously (including the vote period), so meaning
>> 6 weeks
>> between releases.
>>
>
> It seems there have been proposals for monthly, every 6 weeks
> (sesquimonthly?), and bimonthly.
>
>
>> Regarding the time we take for the PR review and merge, I think it's
>> a fair time
>> to give us time to include new improvements and features, but also to
>> fix bugs.
>>
>
> Concrete deadlines can provide good motivation to get around to doing
> reviews, cleaning up PRs, fixing bugs, etc. that you've been meaning to do
> but for whatever reason keep putting off. So I think it's still good
> practice to have some lead time that a release is coming for a chance for
> folks to get stuff in, while still being clear that we're not holding
> things back for new features and if you don't make the cut another one is
> close behind.
>
> Regards
>> JB
>>
>> On 03/01/2018 06:17 PM, Ismaël Mejía wrote:
>> > The average time just in the vote process for Beam since we are out
>> of the
>> > incubator is 17.5 days with an average of 75 days between versions.
>> >
>> > Version  Vote Period   No. days
>> > 2.3.030/01-16/02   17 days  (83 days since last)
>> > 2.2.027/10-25/11   29 days (101 days since last)
>> > 2.1.011/07-16/08   36 days  (92 days since last)
>> > 2.0.008/05-16/058 days  (62 days since last)
>> > 0.6.010/03-15/035 days  (37 days since last)
>> > 0.5.027/01-06/02   10 days
>> >
>> > I think we 

Re: Beam 2.4.0

[Jean-Baptiste Onofré]

Fully agree.

If nobody jumps on it, I will tackle that tomorrow.

Regards
JB

Le 14 mars 2018 à 18:03, à 18:03, Reuven Lax  a écrit:
>Can you add a description and assign a reviewer (using R:). I think
>this
>was basically ready to merge before modulo breaking compilation.
>
>
>On Wed, Mar 14, 2018 at 10:01 AM Romain Manni-Bucau
>
>wrote:
>
>> up, know it missed the 2.4 but can
>> https://github.com/apache/beam/pull/4790 have some love? it really
>makes
>> beam pretty unusable with direct runner,
>> I start to have "// workaround for BEAM-3409" everywhere in my
>codebase
>> which is quite bothering after 3 months :(
>>
>>
>> Romain Manni-Bucau
>> @rmannibucau  |  Blog
>>  | Old Blog
>>  | Github
>>  | LinkedIn
>>  | Book
>>
>
>>
>> 2018-03-06 14:44 GMT+01:00 Romain Manni-Bucau
>:
>>
>>> Hi guys,
>>>
>>> tried to reapply the waitUntilFinish fix - without breaking the
>>> compilation this time ;) - in
>https://github.com/apache/beam/pull/4790,
>>> anyone able to help to review and move that forward?
>>>
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau  |  Blog
>>>  | Old Blog
>>>  | Github
>>>  | LinkedIn
>>>  | Book
>>>
>
>>>
>>> 2018-03-02 9:28 GMT+01:00 Robert Bradshaw :
>>>
 On Fri, Mar 2, 2018 at 12:01 AM Jean-Baptiste Onofré
>
 wrote:

> Hi Ismaël,
>
> that's a good idea to show history.
>
> For me, the vote duration doesn't matter as we are in the release
> process already.
>

 A more relevant duration to track would probably be cut to final
 release. This both measures our investment in the release process,
>as well
 as how behind head is when we finally get the release out.


> The gap between two releases is more significant.


 +1, this is what users see. (To clarify terminology, the "time
>between
 release" is the time between actually releasing x.y and x.y+1 that
>is most
 visible to end users, regardless of intermediate process like
>cutting and
 voting that we have.) Of course this gets thrown off if our
 time-to-prepare-a-release becomes a significant fraction of the
>desired
 time-between-releases.

 And clearly with an average of
> 80 days (~ 3 months) it's two long. The idea is to reduce this
>clearly.
> I
> propose two months previously (including the vote period), so
>meaning 6
> weeks
> between releases.
>

 It seems there have been proposals for monthly, every 6 weeks
 (sesquimonthly?), and bimonthly.


> Regarding the time we take for the PR review and merge, I think
>it's a
> fair time
> to give us time to include new improvements and features, but also
>to
> fix bugs.
>

 Concrete deadlines can provide good motivation to get around to
>doing
 reviews, cleaning up PRs, fixing bugs, etc. that you've been
>meaning to do
 but for whatever reason keep putting off. So I think it's still
>good
 practice to have some lead time that a release is coming for a
>chance for
 folks to get stuff in, while still being clear that we're not
>holding
 things back for new features and if you don't make the cut another
>one is
 close behind.

 Regards
> JB
>
> On 03/01/2018 06:17 PM, Ismaël Mejía wrote:
> > The average time just in the vote process for Beam since we are
>out
> of the
> > incubator is 17.5 days with an average of 75 days between
>versions.
> >
> > Version  Vote Period   No. days
> > 2.3.030/01-16/02   17 days  (83 days since last)
> > 2.2.027/10-25/11   29 days (101 days since last)
> > 2.1.011/07-16/08   36 days  (92 days since last)
> > 2.0.008/05-16/058 days  (62 days since last)
> > 0.6.010/03-15/035 days  (37 days since last)
> > 0.5.027/01-06/02   10 days
> >
> > I think we should have these numbers into account to refine the
> distance between
> > releases. If we want to follow strict time-based releases, what
>we
> can probably
> > refine is how we cut the release so we try to reduce release
>overlaps
> and avoid
> > rushing unnecessarily.
> >
> > Maybe we should follow the proposed 6 weeks for the next release
>like
> this:
> >
> > - 4 weeks let’s say just after succesful vote and then cut the
> release branch.
> > - 1 week to burn 

Re: Beam 2.4.0

[Reuven Lax]

Can you add a description and assign a reviewer (using R:). I think this
was basically ready to merge before modulo breaking compilation.


On Wed, Mar 14, 2018 at 10:01 AM Romain Manni-Bucau 
wrote:

> up, know it missed the 2.4 but can
> https://github.com/apache/beam/pull/4790 have some love? it really makes
> beam pretty unusable with direct runner,
> I start to have "// workaround for BEAM-3409" everywhere in my codebase
> which is quite bothering after 3 months :(
>
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
> 
>
> 2018-03-06 14:44 GMT+01:00 Romain Manni-Bucau :
>
>> Hi guys,
>>
>> tried to reapply the waitUntilFinish fix - without breaking the
>> compilation this time ;) - in https://github.com/apache/beam/pull/4790,
>> anyone able to help to review and move that forward?
>>
>>
>> Romain Manni-Bucau
>> @rmannibucau  |  Blog
>>  | Old Blog
>>  | Github
>>  | LinkedIn
>>  | Book
>> 
>>
>> 2018-03-02 9:28 GMT+01:00 Robert Bradshaw :
>>
>>> On Fri, Mar 2, 2018 at 12:01 AM Jean-Baptiste Onofré 
>>> wrote:
>>>
 Hi Ismaël,

 that's a good idea to show history.

 For me, the vote duration doesn't matter as we are in the release
 process already.

>>>
>>> A more relevant duration to track would probably be cut to final
>>> release. This both measures our investment in the release process, as well
>>> as how behind head is when we finally get the release out.
>>>
>>>
 The gap between two releases is more significant.
>>>
>>>
>>> +1, this is what users see. (To clarify terminology, the "time between
>>> release" is the time between actually releasing x.y and x.y+1 that is most
>>> visible to end users, regardless of intermediate process like cutting and
>>> voting that we have.) Of course this gets thrown off if our
>>> time-to-prepare-a-release becomes a significant fraction of the desired
>>> time-between-releases.
>>>
>>> And clearly with an average of
 80 days (~ 3 months) it's two long. The idea is to reduce this clearly.
 I
 propose two months previously (including the vote period), so meaning 6
 weeks
 between releases.

>>>
>>> It seems there have been proposals for monthly, every 6 weeks
>>> (sesquimonthly?), and bimonthly.
>>>
>>>
 Regarding the time we take for the PR review and merge, I think it's a
 fair time
 to give us time to include new improvements and features, but also to
 fix bugs.

>>>
>>> Concrete deadlines can provide good motivation to get around to doing
>>> reviews, cleaning up PRs, fixing bugs, etc. that you've been meaning to do
>>> but for whatever reason keep putting off. So I think it's still good
>>> practice to have some lead time that a release is coming for a chance for
>>> folks to get stuff in, while still being clear that we're not holding
>>> things back for new features and if you don't make the cut another one is
>>> close behind.
>>>
>>> Regards
 JB

 On 03/01/2018 06:17 PM, Ismaël Mejía wrote:
 > The average time just in the vote process for Beam since we are out
 of the
 > incubator is 17.5 days with an average of 75 days between versions.
 >
 > Version  Vote Period   No. days
 > 2.3.030/01-16/02   17 days  (83 days since last)
 > 2.2.027/10-25/11   29 days (101 days since last)
 > 2.1.011/07-16/08   36 days  (92 days since last)
 > 2.0.008/05-16/058 days  (62 days since last)
 > 0.6.010/03-15/035 days  (37 days since last)
 > 0.5.027/01-06/02   10 days
 >
 > I think we should have these numbers into account to refine the
 distance between
 > releases. If we want to follow strict time-based releases, what we
 can probably
 > refine is how we cut the release so we try to reduce release overlaps
 and avoid
 > rushing unnecessarily.
 >
 > Maybe we should follow the proposed 6 weeks for the next release like
 this:
 >
 > - 4 weeks let’s say just after succesful vote and then cut the
 release branch.
 > - 1 week to burn the blocker list (good to have ones that don’t make
 will be
 >   moved to the next release).
 > - 1 week for the vote + RCs (in case the vote takes longer at least
 the overlap
 >   between vote + next dev cycle will be smaller).
 >
 > If we do this for the next cycle we 

Re: Beam 2.4.0

[Romain Manni-Bucau]

up, know it missed the 2.4 but can https://github.com/apache/beam/pull/4790
have some love? it really makes beam pretty unusable with direct runner,
I start to have "// workaround for BEAM-3409" everywhere in my codebase
which is quite bothering after 3 months :(


Romain Manni-Bucau
@rmannibucau  |  Blog
 | Old Blog
 | Github  |
LinkedIn  | Book


2018-03-06 14:44 GMT+01:00 Romain Manni-Bucau :

> Hi guys,
>
> tried to reapply the waitUntilFinish fix - without breaking the
> compilation this time ;) - in https://github.com/apache/beam/pull/4790,
> anyone able to help to review and move that forward?
>
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
> 
>
> 2018-03-02 9:28 GMT+01:00 Robert Bradshaw :
>
>> On Fri, Mar 2, 2018 at 12:01 AM Jean-Baptiste Onofré 
>> wrote:
>>
>>> Hi Ismaël,
>>>
>>> that's a good idea to show history.
>>>
>>> For me, the vote duration doesn't matter as we are in the release
>>> process already.
>>>
>>
>> A more relevant duration to track would probably be cut to final release.
>> This both measures our investment in the release process, as well as how
>> behind head is when we finally get the release out.
>>
>>
>>> The gap between two releases is more significant.
>>
>>
>> +1, this is what users see. (To clarify terminology, the "time between
>> release" is the time between actually releasing x.y and x.y+1 that is most
>> visible to end users, regardless of intermediate process like cutting and
>> voting that we have.) Of course this gets thrown off if our
>> time-to-prepare-a-release becomes a significant fraction of the desired
>> time-between-releases.
>>
>> And clearly with an average of
>>> 80 days (~ 3 months) it's two long. The idea is to reduce this clearly. I
>>> propose two months previously (including the vote period), so meaning 6
>>> weeks
>>> between releases.
>>>
>>
>> It seems there have been proposals for monthly, every 6 weeks
>> (sesquimonthly?), and bimonthly.
>>
>>
>>> Regarding the time we take for the PR review and merge, I think it's a
>>> fair time
>>> to give us time to include new improvements and features, but also to
>>> fix bugs.
>>>
>>
>> Concrete deadlines can provide good motivation to get around to doing
>> reviews, cleaning up PRs, fixing bugs, etc. that you've been meaning to do
>> but for whatever reason keep putting off. So I think it's still good
>> practice to have some lead time that a release is coming for a chance for
>> folks to get stuff in, while still being clear that we're not holding
>> things back for new features and if you don't make the cut another one is
>> close behind.
>>
>> Regards
>>> JB
>>>
>>> On 03/01/2018 06:17 PM, Ismaël Mejía wrote:
>>> > The average time just in the vote process for Beam since we are out of
>>> the
>>> > incubator is 17.5 days with an average of 75 days between versions.
>>> >
>>> > Version  Vote Period   No. days
>>> > 2.3.030/01-16/02   17 days  (83 days since last)
>>> > 2.2.027/10-25/11   29 days (101 days since last)
>>> > 2.1.011/07-16/08   36 days  (92 days since last)
>>> > 2.0.008/05-16/058 days  (62 days since last)
>>> > 0.6.010/03-15/035 days  (37 days since last)
>>> > 0.5.027/01-06/02   10 days
>>> >
>>> > I think we should have these numbers into account to refine the
>>> distance between
>>> > releases. If we want to follow strict time-based releases, what we can
>>> probably
>>> > refine is how we cut the release so we try to reduce release overlaps
>>> and avoid
>>> > rushing unnecessarily.
>>> >
>>> > Maybe we should follow the proposed 6 weeks for the next release like
>>> this:
>>> >
>>> > - 4 weeks let’s say just after succesful vote and then cut the release
>>> branch.
>>> > - 1 week to burn the blocker list (good to have ones that don’t make
>>> will be
>>> >   moved to the next release).
>>> > - 1 week for the vote + RCs (in case the vote takes longer at least
>>> the overlap
>>> >   between vote + next dev cycle will be smaller).
>>> >
>>> > If we do this for the next cycle we will have a 6 week ‘dev’ period
>>> and then we
>>> > will have optimistically an average of 2 weeks of ‘releasing’ and 6
>>> weeks ‘dev’
>>> > cycles.
>>> >
>>> > On Thu, Mar 1, 2018 at 6:46 AM, Jean-Baptiste Onofré 
>>> wrote:
>>> >> About BEAM-3409, I did a review yesterday and it looks good to me. We
>>> are
>>> >> waiting for Thomas' 

Re: Beam 2.4.0

[Romain Manni-Bucau]

Hi guys,

tried to reapply the waitUntilFinish fix - without breaking the compilation
this time ;) - in https://github.com/apache/beam/pull/4790, anyone able to
help to review and move that forward?


Romain Manni-Bucau
@rmannibucau  |  Blog
 | Old Blog
 | Github  |
LinkedIn  | Book


2018-03-02 9:28 GMT+01:00 Robert Bradshaw :

> On Fri, Mar 2, 2018 at 12:01 AM Jean-Baptiste Onofré 
> wrote:
>
>> Hi Ismaël,
>>
>> that's a good idea to show history.
>>
>> For me, the vote duration doesn't matter as we are in the release process
>> already.
>>
>
> A more relevant duration to track would probably be cut to final release.
> This both measures our investment in the release process, as well as how
> behind head is when we finally get the release out.
>
>
>> The gap between two releases is more significant.
>
>
> +1, this is what users see. (To clarify terminology, the "time between
> release" is the time between actually releasing x.y and x.y+1 that is most
> visible to end users, regardless of intermediate process like cutting and
> voting that we have.) Of course this gets thrown off if our
> time-to-prepare-a-release becomes a significant fraction of the desired
> time-between-releases.
>
> And clearly with an average of
>> 80 days (~ 3 months) it's two long. The idea is to reduce this clearly. I
>> propose two months previously (including the vote period), so meaning 6
>> weeks
>> between releases.
>>
>
> It seems there have been proposals for monthly, every 6 weeks
> (sesquimonthly?), and bimonthly.
>
>
>> Regarding the time we take for the PR review and merge, I think it's a
>> fair time
>> to give us time to include new improvements and features, but also to fix
>> bugs.
>>
>
> Concrete deadlines can provide good motivation to get around to doing
> reviews, cleaning up PRs, fixing bugs, etc. that you've been meaning to do
> but for whatever reason keep putting off. So I think it's still good
> practice to have some lead time that a release is coming for a chance for
> folks to get stuff in, while still being clear that we're not holding
> things back for new features and if you don't make the cut another one is
> close behind.
>
> Regards
>> JB
>>
>> On 03/01/2018 06:17 PM, Ismaël Mejía wrote:
>> > The average time just in the vote process for Beam since we are out of
>> the
>> > incubator is 17.5 days with an average of 75 days between versions.
>> >
>> > Version  Vote Period   No. days
>> > 2.3.030/01-16/02   17 days  (83 days since last)
>> > 2.2.027/10-25/11   29 days (101 days since last)
>> > 2.1.011/07-16/08   36 days  (92 days since last)
>> > 2.0.008/05-16/058 days  (62 days since last)
>> > 0.6.010/03-15/035 days  (37 days since last)
>> > 0.5.027/01-06/02   10 days
>> >
>> > I think we should have these numbers into account to refine the
>> distance between
>> > releases. If we want to follow strict time-based releases, what we can
>> probably
>> > refine is how we cut the release so we try to reduce release overlaps
>> and avoid
>> > rushing unnecessarily.
>> >
>> > Maybe we should follow the proposed 6 weeks for the next release like
>> this:
>> >
>> > - 4 weeks let’s say just after succesful vote and then cut the release
>> branch.
>> > - 1 week to burn the blocker list (good to have ones that don’t make
>> will be
>> >   moved to the next release).
>> > - 1 week for the vote + RCs (in case the vote takes longer at least the
>> overlap
>> >   between vote + next dev cycle will be smaller).
>> >
>> > If we do this for the next cycle we will have a 6 week ‘dev’ period and
>> then we
>> > will have optimistically an average of 2 weeks of ‘releasing’ and 6
>> weeks ‘dev’
>> > cycles.
>> >
>> > On Thu, Mar 1, 2018 at 6:46 AM, Jean-Baptiste Onofré 
>> wrote:
>> >> About BEAM-3409, I did a review yesterday and it looks good to me. We
>> are
>> >> waiting for Thomas' feedback.
>> >>
>> >> Regards
>> >> JB
>> >> Le 1 mars 2018, à 09:38, Robert Bradshaw  a
>> écrit:
>> >>>
>> >>> Looking at the burn-down list, we have 5 remaining issues. None of
>> these
>> >>> are blockers, but all look like they're really close (reviewed, review
>> >>> comments were addressed, waiting for a final LGTM). Specifically:
>> >>>
>> >>> BEAM-3409 (teardown issues): Thomas Groh had some concerns, could you
>> >>> verify they have all been addressed?
>> >>> BEAM-3479 (DoFn classloader  regression test): Kenn Knowles had minor
>> >>> comments, looks like they were addressed, could you confirm?
>> >>> BEAM-3735 (Missing gaming release archetypes): Lukasz Cwik had minor
>> >>> comments, looks like they were addressed, could you confirm?
>> >>> BEAM-3611 (KafkaIO.java 

Re: Beam 2.4.0

[Robert Bradshaw]

On Fri, Mar 2, 2018 at 12:01 AM Jean-Baptiste Onofré 
wrote:

> Hi Ismaël,
>
> that's a good idea to show history.
>
> For me, the vote duration doesn't matter as we are in the release process
> already.
>

A more relevant duration to track would probably be cut to final release.
This both measures our investment in the release process, as well as how
behind head is when we finally get the release out.


> The gap between two releases is more significant.


+1, this is what users see. (To clarify terminology, the "time between
release" is the time between actually releasing x.y and x.y+1 that is most
visible to end users, regardless of intermediate process like cutting and
voting that we have.) Of course this gets thrown off if our
time-to-prepare-a-release becomes a significant fraction of the desired
time-between-releases.

And clearly with an average of
> 80 days (~ 3 months) it's two long. The idea is to reduce this clearly. I
> propose two months previously (including the vote period), so meaning 6
> weeks
> between releases.
>

It seems there have been proposals for monthly, every 6 weeks
(sesquimonthly?), and bimonthly.


> Regarding the time we take for the PR review and merge, I think it's a
> fair time
> to give us time to include new improvements and features, but also to fix
> bugs.
>

Concrete deadlines can provide good motivation to get around to doing
reviews, cleaning up PRs, fixing bugs, etc. that you've been meaning to do
but for whatever reason keep putting off. So I think it's still good
practice to have some lead time that a release is coming for a chance for
folks to get stuff in, while still being clear that we're not holding
things back for new features and if you don't make the cut another one is
close behind.

Regards
> JB
>
> On 03/01/2018 06:17 PM, Ismaël Mejía wrote:
> > The average time just in the vote process for Beam since we are out of
> the
> > incubator is 17.5 days with an average of 75 days between versions.
> >
> > Version  Vote Period   No. days
> > 2.3.030/01-16/02   17 days  (83 days since last)
> > 2.2.027/10-25/11   29 days (101 days since last)
> > 2.1.011/07-16/08   36 days  (92 days since last)
> > 2.0.008/05-16/058 days  (62 days since last)
> > 0.6.010/03-15/035 days  (37 days since last)
> > 0.5.027/01-06/02   10 days
> >
> > I think we should have these numbers into account to refine the distance
> between
> > releases. If we want to follow strict time-based releases, what we can
> probably
> > refine is how we cut the release so we try to reduce release overlaps
> and avoid
> > rushing unnecessarily.
> >
> > Maybe we should follow the proposed 6 weeks for the next release like
> this:
> >
> > - 4 weeks let’s say just after succesful vote and then cut the release
> branch.
> > - 1 week to burn the blocker list (good to have ones that don’t make
> will be
> >   moved to the next release).
> > - 1 week for the vote + RCs (in case the vote takes longer at least the
> overlap
> >   between vote + next dev cycle will be smaller).
> >
> > If we do this for the next cycle we will have a 6 week ‘dev’ period and
> then we
> > will have optimistically an average of 2 weeks of ‘releasing’ and 6
> weeks ‘dev’
> > cycles.
> >
> > On Thu, Mar 1, 2018 at 6:46 AM, Jean-Baptiste Onofré 
> wrote:
> >> About BEAM-3409, I did a review yesterday and it looks good to me. We
> are
> >> waiting for Thomas' feedback.
> >>
> >> Regards
> >> JB
> >> Le 1 mars 2018, à 09:38, Robert Bradshaw  a écrit:
> >>>
> >>> Looking at the burn-down list, we have 5 remaining issues. None of
> these
> >>> are blockers, but all look like they're really close (reviewed, review
> >>> comments were addressed, waiting for a final LGTM). Specifically:
> >>>
> >>> BEAM-3409 (teardown issues): Thomas Groh had some concerns, could you
> >>> verify they have all been addressed?
> >>> BEAM-3479 (DoFn classloader  regression test): Kenn Knowles had minor
> >>> comments, looks like they were addressed, could you confirm?
> >>> BEAM-3735 (Missing gaming release archetypes): Lukasz Cwik had minor
> >>> comments, looks like they were addressed, could you confirm?
> >>> BEAM-3611 (KafkaIO.java splitting): Looks like this was resolved.
> >>> BEAM-3762 (unlimited JCE for Dataflow Worker): LGTM pending (currently
> >>> running) tests passing.
> >>>
> >>> Let's see how many of these we can get in by, say, noon PST tomorrow.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Wed, Feb 28, 2018 at 9:26 PM Robert Bradshaw < rober...@google.com>
> >>> wrote:
> 
>  I tend to fall into the "release early, release often" camp in
> general,
>  but for this one I'm particularly anxious to get the faster Python
> direct
>  runner out in the hands of TFT/TFX users (and in particular have an
> eye on
>  https://www.tensorflow.org/dev-summit/ which I think can be a
> healthy source
>  of Beam users).
> 
> 
>  On Wed, 

Re: Beam 2.4.0

[Jean-Baptiste Onofré]

Hi Ismaël,

that's a good idea to show history.

For me, the vote duration doesn't matter as we are in the release process 
already.

The gap between two releases is more significant. And clearly with an average of
80 days (~ 3 months) it's two long. The idea is to reduce this clearly. I
propose two months previously (including the vote period), so meaning 6 weeks
between releases.

Regarding the time we take for the PR review and merge, I think it's a fair time
to give us time to include new improvements and features, but also to fix bugs.

Regards
JB

On 03/01/2018 06:17 PM, Ismaël Mejía wrote:
> The average time just in the vote process for Beam since we are out of the
> incubator is 17.5 days with an average of 75 days between versions.
> 
> Version  Vote Period   No. days
> 2.3.030/01-16/02   17 days  (83 days since last)
> 2.2.027/10-25/11   29 days (101 days since last)
> 2.1.011/07-16/08   36 days  (92 days since last)
> 2.0.008/05-16/058 days  (62 days since last)
> 0.6.010/03-15/035 days  (37 days since last)
> 0.5.027/01-06/02   10 days
> 
> I think we should have these numbers into account to refine the distance 
> between
> releases. If we want to follow strict time-based releases, what we can 
> probably
> refine is how we cut the release so we try to reduce release overlaps and 
> avoid
> rushing unnecessarily.
> 
> Maybe we should follow the proposed 6 weeks for the next release like this:
> 
> - 4 weeks let’s say just after succesful vote and then cut the release branch.
> - 1 week to burn the blocker list (good to have ones that don’t make will be
>   moved to the next release).
> - 1 week for the vote + RCs (in case the vote takes longer at least the 
> overlap
>   between vote + next dev cycle will be smaller).
> 
> If we do this for the next cycle we will have a 6 week ‘dev’ period and then 
> we
> will have optimistically an average of 2 weeks of ‘releasing’ and 6 weeks 
> ‘dev’
> cycles.
> 
> On Thu, Mar 1, 2018 at 6:46 AM, Jean-Baptiste Onofré  
> wrote:
>> About BEAM-3409, I did a review yesterday and it looks good to me. We are
>> waiting for Thomas' feedback.
>>
>> Regards
>> JB
>> Le 1 mars 2018, à 09:38, Robert Bradshaw  a écrit:
>>>
>>> Looking at the burn-down list, we have 5 remaining issues. None of these
>>> are blockers, but all look like they're really close (reviewed, review
>>> comments were addressed, waiting for a final LGTM). Specifically:
>>>
>>> BEAM-3409 (teardown issues): Thomas Groh had some concerns, could you
>>> verify they have all been addressed?
>>> BEAM-3479 (DoFn classloader  regression test): Kenn Knowles had minor
>>> comments, looks like they were addressed, could you confirm?
>>> BEAM-3735 (Missing gaming release archetypes): Lukasz Cwik had minor
>>> comments, looks like they were addressed, could you confirm?
>>> BEAM-3611 (KafkaIO.java splitting): Looks like this was resolved.
>>> BEAM-3762 (unlimited JCE for Dataflow Worker): LGTM pending (currently
>>> running) tests passing.
>>>
>>> Let's see how many of these we can get in by, say, noon PST tomorrow.
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Feb 28, 2018 at 9:26 PM Robert Bradshaw < rober...@google.com>
>>> wrote:

 I tend to fall into the "release early, release often" camp in general,
 but for this one I'm particularly anxious to get the faster Python direct
 runner out in the hands of TFT/TFX users (and in particular have an eye on
 https://www.tensorflow.org/dev-summit/ which I think can be a healthy 
 source
 of Beam users).


 On Wed, Feb 28, 2018 at 7:01 PM Jean-Baptiste Onofré < j...@nanthrax.net>
 wrote:
>
> Hi Gus,
>
> Thanks for the update, it makes sense.
>
> Regards
> JB
>
> On 03/01/2018 02:59 AM, Konstantinos Katsiapis wrote:
>> Hi Jean-Baptiste,
>>
>> I can speak from the perspective of tf.transform
>> < https://github.com/tensorflow/transform> (TFT) in particular and TFX
>> < https://research.google.com/pubs/pub46484.html> libs in general, in
>> case it is
>> useful.
>>
>> TFX distributed computation has 2 "large" dependencies, namely
>> TensorFlow and
>> Apache Beam, each on their own release schedule.
>> As such, releasing of new TFX functionality often (but not always)
>> depends on
>> (and is blocked by) releases of *both* TensorFlow *and* Apache Beam.
>>
>> Synchronizing releases across such large projects and organizations is
>> likely
>> hard, so from our perspective having *frequent* releases of Tensorflow
>> or Apache
>> Beam (and better yet both) decreases the time for which we are blocked
>> on
>> releasing our features.
>>
>> In light of this, I would vote for more frequent releases in general,
>> and for a
>> Beam 2.4 release soon in particular (as TFT 0.6 depends on it).
>>
>> Thanks,
>> Gus
>>
>> On 

Re: Beam 2.4.0

[Kenneth Knowles]

The features/bugfixes included in a release are determined by the time
between cutting release branches. So I'd focus on that cadence (outside of
special requests).

Kenn


On Thu, Mar 1, 2018 at 12:33 PM, Robert Bradshaw 
wrote:

> On Thu, Mar 1, 2018 at 9:17 AM Ismaël Mejía  wrote:
>
>> The average time just in the vote process for Beam since we are out of the
>> incubator is 17.5 days with an average of 75 days between versions.
>>
>
> Good thought to look at history. I think there's general consensus that
> this is longer than we would like.
>
>
>> Version  Vote Period   No. days
>> 2.3.030/01-16/02   17 days  (83 days since last)
>> 2.2.027/10-25/11   29 days (101 days since last)
>> 2.1.011/07-16/08   36 days  (92 days since last)
>> 2.0.008/05-16/058 days  (62 days since last)
>> 0.6.010/03-15/035 days  (37 days since last)
>> 0.5.027/01-06/02   10 days
>>
>> I think we should have these numbers into account to refine the distance
>> between
>> releases. If we want to follow strict time-based releases, what we can
>> probably
>> refine is how we cut the release so we try to reduce release overlaps and
>> avoid
>> rushing unnecessarily.
>>
>> Maybe we should follow the proposed 6 weeks for the next release like
>> this:
>>
>> - 4 weeks let’s say just after succesful vote and then cut the release
>> branch.
>> - 1 week to burn the blocker list (good to have ones that don’t make will
>> be
>>   moved to the next release).
>> - 1 week for the vote + RCs (in case the vote takes longer at least the
>> overlap
>>   between vote + next dev cycle will be smaller).
>>
>> If we do this for the next cycle we will have a 6 week ‘dev’ period and
>> then we
>> will have optimistically an average of 2 weeks of ‘releasing’ and 6 weeks
>> ‘dev’
>> cycles.
>>
>
> 1 week vote seems optimistic. On the other hand, the reason to have a
> release branch is so that dev work can continue during an ongoing release.
>
>
>> On Thu, Mar 1, 2018 at 6:46 AM, Jean-Baptiste Onofré 
>> wrote:
>> > About BEAM-3409, I did a review yesterday and it looks good to me. We
>> are
>> > waiting for Thomas' feedback.
>> >
>> > Regards
>> > JB
>> > Le 1 mars 2018, à 09:38, Robert Bradshaw  a écrit:
>> >>
>> >> Looking at the burn-down list, we have 5 remaining issues. None of
>> these
>> >> are blockers, but all look like they're really close (reviewed, review
>> >> comments were addressed, waiting for a final LGTM). Specifically:
>> >>
>> >> BEAM-3409 (teardown issues): Thomas Groh had some concerns, could you
>> >> verify they have all been addressed?
>> >> BEAM-3479 (DoFn classloader  regression test): Kenn Knowles had minor
>> >> comments, looks like they were addressed, could you confirm?
>> >> BEAM-3735 (Missing gaming release archetypes): Lukasz Cwik had minor
>> >> comments, looks like they were addressed, could you confirm?
>> >> BEAM-3611 (KafkaIO.java splitting): Looks like this was resolved.
>> >> BEAM-3762 (unlimited JCE for Dataflow Worker): LGTM pending (currently
>> >> running) tests passing.
>> >>
>> >> Let's see how many of these we can get in by, say, noon PST tomorrow.
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Wed, Feb 28, 2018 at 9:26 PM Robert Bradshaw < rober...@google.com>
>> >> wrote:
>> >>>
>> >>> I tend to fall into the "release early, release often" camp in
>> general,
>> >>> but for this one I'm particularly anxious to get the faster Python
>> direct
>> >>> runner out in the hands of TFT/TFX users (and in particular have an
>> eye on
>> >>> https://www.tensorflow.org/dev-summit/ which I think can be a
>> healthy source
>> >>> of Beam users).
>> >>>
>> >>>
>> >>> On Wed, Feb 28, 2018 at 7:01 PM Jean-Baptiste Onofré <
>> j...@nanthrax.net>
>> >>> wrote:
>> 
>>  Hi Gus,
>> 
>>  Thanks for the update, it makes sense.
>> 
>>  Regards
>>  JB
>> 
>>  On 03/01/2018 02:59 AM, Konstantinos Katsiapis wrote:
>>  > Hi Jean-Baptiste,
>>  >
>>  > I can speak from the perspective of tf.transform
>>  > < https://github.com/tensorflow/transform> (TFT) in particular
>> and TFX
>>  > < https://research.google.com/pubs/pub46484.html> libs in
>> general, in
>>  > case it is
>>  > useful.
>>  >
>>  > TFX distributed computation has 2 "large" dependencies, namely
>>  > TensorFlow and
>>  > Apache Beam, each on their own release schedule.
>>  > As such, releasing of new TFX functionality often (but not always)
>>  > depends on
>>  > (and is blocked by) releases of *both* TensorFlow *and* Apache
>> Beam.
>>  >
>>  > Synchronizing releases across such large projects and
>> organizations is
>>  > likely
>>  > hard, so from our perspective having *frequent* releases of
>> Tensorflow
>>  > or Apache
>>  > Beam (and better yet both) decreases the time for which we are
>> blocked
>>  > on
>>  > releasing our features.

Re: Beam 2.4.0

[Robert Bradshaw]

On Thu, Mar 1, 2018 at 9:17 AM Ismaël Mejía  wrote:

> The average time just in the vote process for Beam since we are out of the
> incubator is 17.5 days with an average of 75 days between versions.
>

Good thought to look at history. I think there's general consensus that
this is longer than we would like.


> Version  Vote Period   No. days
> 2.3.030/01-16/02   17 days  (83 days since last)
> 2.2.027/10-25/11   29 days (101 days since last)
> 2.1.011/07-16/08   36 days  (92 days since last)
> 2.0.008/05-16/058 days  (62 days since last)
> 0.6.010/03-15/035 days  (37 days since last)
> 0.5.027/01-06/02   10 days
>
> I think we should have these numbers into account to refine the distance
> between
> releases. If we want to follow strict time-based releases, what we can
> probably
> refine is how we cut the release so we try to reduce release overlaps and
> avoid
> rushing unnecessarily.
>
> Maybe we should follow the proposed 6 weeks for the next release like this:
>
> - 4 weeks let’s say just after succesful vote and then cut the release
> branch.
> - 1 week to burn the blocker list (good to have ones that don’t make will
> be
>   moved to the next release).
> - 1 week for the vote + RCs (in case the vote takes longer at least the
> overlap
>   between vote + next dev cycle will be smaller).
>
> If we do this for the next cycle we will have a 6 week ‘dev’ period and
> then we
> will have optimistically an average of 2 weeks of ‘releasing’ and 6 weeks
> ‘dev’
> cycles.
>

1 week vote seems optimistic. On the other hand, the reason to have a
release branch is so that dev work can continue during an ongoing release.


> On Thu, Mar 1, 2018 at 6:46 AM, Jean-Baptiste Onofré 
> wrote:
> > About BEAM-3409, I did a review yesterday and it looks good to me. We are
> > waiting for Thomas' feedback.
> >
> > Regards
> > JB
> > Le 1 mars 2018, à 09:38, Robert Bradshaw  a écrit:
> >>
> >> Looking at the burn-down list, we have 5 remaining issues. None of these
> >> are blockers, but all look like they're really close (reviewed, review
> >> comments were addressed, waiting for a final LGTM). Specifically:
> >>
> >> BEAM-3409 (teardown issues): Thomas Groh had some concerns, could you
> >> verify they have all been addressed?
> >> BEAM-3479 (DoFn classloader  regression test): Kenn Knowles had minor
> >> comments, looks like they were addressed, could you confirm?
> >> BEAM-3735 (Missing gaming release archetypes): Lukasz Cwik had minor
> >> comments, looks like they were addressed, could you confirm?
> >> BEAM-3611 (KafkaIO.java splitting): Looks like this was resolved.
> >> BEAM-3762 (unlimited JCE for Dataflow Worker): LGTM pending (currently
> >> running) tests passing.
> >>
> >> Let's see how many of these we can get in by, say, noon PST tomorrow.
> >>
> >>
> >>
> >>
> >>
> >> On Wed, Feb 28, 2018 at 9:26 PM Robert Bradshaw < rober...@google.com>
> >> wrote:
> >>>
> >>> I tend to fall into the "release early, release often" camp in general,
> >>> but for this one I'm particularly anxious to get the faster Python
> direct
> >>> runner out in the hands of TFT/TFX users (and in particular have an
> eye on
> >>> https://www.tensorflow.org/dev-summit/ which I think can be a healthy
> source
> >>> of Beam users).
> >>>
> >>>
> >>> On Wed, Feb 28, 2018 at 7:01 PM Jean-Baptiste Onofré < j...@nanthrax.net
> >
> >>> wrote:
> 
>  Hi Gus,
> 
>  Thanks for the update, it makes sense.
> 
>  Regards
>  JB
> 
>  On 03/01/2018 02:59 AM, Konstantinos Katsiapis wrote:
>  > Hi Jean-Baptiste,
>  >
>  > I can speak from the perspective of tf.transform
>  > < https://github.com/tensorflow/transform> (TFT) in particular and
> TFX
>  > < https://research.google.com/pubs/pub46484.html> libs in general,
> in
>  > case it is
>  > useful.
>  >
>  > TFX distributed computation has 2 "large" dependencies, namely
>  > TensorFlow and
>  > Apache Beam, each on their own release schedule.
>  > As such, releasing of new TFX functionality often (but not always)
>  > depends on
>  > (and is blocked by) releases of *both* TensorFlow *and* Apache Beam.
>  >
>  > Synchronizing releases across such large projects and organizations
> is
>  > likely
>  > hard, so from our perspective having *frequent* releases of
> Tensorflow
>  > or Apache
>  > Beam (and better yet both) decreases the time for which we are
> blocked
>  > on
>  > releasing our features.
>  >
>  > In light of this, I would vote for more frequent releases in
> general,
>  > and for a
>  > Beam 2.4 release soon in particular (as TFT 0.6 depends on it).
>  >
>  > Thanks,
>  > Gus
>  >
>  > On Tue, Feb 27, 2018 at 10:29 PM, Jean-Baptiste Onofré <
>  > j...@nanthrax.net
>  > > wrote:
>  >
>  > By the way, if third party projects 

Re: Beam 2.4.0

[Ismaël Mejía]

The average time just in the vote process for Beam since we are out of the
incubator is 17.5 days with an average of 75 days between versions.

Version  Vote Period   No. days
2.3.030/01-16/02   17 days  (83 days since last)
2.2.027/10-25/11   29 days (101 days since last)
2.1.011/07-16/08   36 days  (92 days since last)
2.0.008/05-16/058 days  (62 days since last)
0.6.010/03-15/035 days  (37 days since last)
0.5.027/01-06/02   10 days

I think we should have these numbers into account to refine the distance between
releases. If we want to follow strict time-based releases, what we can probably
refine is how we cut the release so we try to reduce release overlaps and avoid
rushing unnecessarily.

Maybe we should follow the proposed 6 weeks for the next release like this:

- 4 weeks let’s say just after succesful vote and then cut the release branch.
- 1 week to burn the blocker list (good to have ones that don’t make will be
  moved to the next release).
- 1 week for the vote + RCs (in case the vote takes longer at least the overlap
  between vote + next dev cycle will be smaller).

If we do this for the next cycle we will have a 6 week ‘dev’ period and then we
will have optimistically an average of 2 weeks of ‘releasing’ and 6 weeks ‘dev’
cycles.

On Thu, Mar 1, 2018 at 6:46 AM, Jean-Baptiste Onofré  wrote:
> About BEAM-3409, I did a review yesterday and it looks good to me. We are
> waiting for Thomas' feedback.
>
> Regards
> JB
> Le 1 mars 2018, à 09:38, Robert Bradshaw  a écrit:
>>
>> Looking at the burn-down list, we have 5 remaining issues. None of these
>> are blockers, but all look like they're really close (reviewed, review
>> comments were addressed, waiting for a final LGTM). Specifically:
>>
>> BEAM-3409 (teardown issues): Thomas Groh had some concerns, could you
>> verify they have all been addressed?
>> BEAM-3479 (DoFn classloader  regression test): Kenn Knowles had minor
>> comments, looks like they were addressed, could you confirm?
>> BEAM-3735 (Missing gaming release archetypes): Lukasz Cwik had minor
>> comments, looks like they were addressed, could you confirm?
>> BEAM-3611 (KafkaIO.java splitting): Looks like this was resolved.
>> BEAM-3762 (unlimited JCE for Dataflow Worker): LGTM pending (currently
>> running) tests passing.
>>
>> Let's see how many of these we can get in by, say, noon PST tomorrow.
>>
>>
>>
>>
>>
>> On Wed, Feb 28, 2018 at 9:26 PM Robert Bradshaw < rober...@google.com>
>> wrote:
>>>
>>> I tend to fall into the "release early, release often" camp in general,
>>> but for this one I'm particularly anxious to get the faster Python direct
>>> runner out in the hands of TFT/TFX users (and in particular have an eye on
>>> https://www.tensorflow.org/dev-summit/ which I think can be a healthy source
>>> of Beam users).
>>>
>>>
>>> On Wed, Feb 28, 2018 at 7:01 PM Jean-Baptiste Onofré < j...@nanthrax.net>
>>> wrote:

 Hi Gus,

 Thanks for the update, it makes sense.

 Regards
 JB

 On 03/01/2018 02:59 AM, Konstantinos Katsiapis wrote:
 > Hi Jean-Baptiste,
 >
 > I can speak from the perspective of tf.transform
 > < https://github.com/tensorflow/transform> (TFT) in particular and TFX
 > < https://research.google.com/pubs/pub46484.html> libs in general, in
 > case it is
 > useful.
 >
 > TFX distributed computation has 2 "large" dependencies, namely
 > TensorFlow and
 > Apache Beam, each on their own release schedule.
 > As such, releasing of new TFX functionality often (but not always)
 > depends on
 > (and is blocked by) releases of *both* TensorFlow *and* Apache Beam.
 >
 > Synchronizing releases across such large projects and organizations is
 > likely
 > hard, so from our perspective having *frequent* releases of Tensorflow
 > or Apache
 > Beam (and better yet both) decreases the time for which we are blocked
 > on
 > releasing our features.
 >
 > In light of this, I would vote for more frequent releases in general,
 > and for a
 > Beam 2.4 release soon in particular (as TFT 0.6 depends on it).
 >
 > Thanks,
 > Gus
 >
 > On Tue, Feb 27, 2018 at 10:29 PM, Jean-Baptiste Onofré <
 > j...@nanthrax.net
 > > wrote:
 >
 > By the way, if third party projects based on Beam (Google
 > Dataflow, Talend
 > DataStream, and others) need a release (including some features),
 > it's better to
 >  clearly state this on the mailing list.
 >
 > At Apache Karaf, I have lot of projects based on it (OpenDaylight,
 > OpenHAB,
 > Websphere,  ...). They just ask for the release schedule and they
 > align with
 > these release. As a best effort, I'm always trying to move fast
 > when a release
 > is asked.
 >
 > So, if 2.4.0 is required by third party, no problem to 

Re: Beam 2.4.0

[Jean-Baptiste Onofré]

About BEAM-3409, I did a review yesterday and it looks good to me. We are 
waiting for Thomas' feedback.

Regards
JB

Le 1 mars 2018 à 09:38, à 09:38, Robert Bradshaw  a écrit:
>Looking at the burn-down list, we have 5 remaining issues. None of
>these
>are blockers, but all look like they're really close (reviewed, review
>comments were addressed, waiting for a final LGTM). Specifically:
>
>BEAM-3409 (teardown issues): Thomas Groh had some concerns, could you
>verify they have all been addressed?
>BEAM-3479 (DoFn classloader regression test): Kenn Knowles had minor
>comments, looks like they were addressed, could you confirm?
>BEAM-3735 (Missing gaming release archetypes): Lukasz Cwik had minor
>comments, looks like they were addressed, could you confirm?
>BEAM-3611 (KafkaIO.java splitting): Looks like this was resolved.
>BEAM-3762 (unlimited JCE for Dataflow Worker): LGTM pending (currently
>running) tests passing.
>
>Let's see how many of these we can get in by, say, noon PST tomorrow.
>
>
>
>
>
>On Wed, Feb 28, 2018 at 9:26 PM Robert Bradshaw 
>wrote:
>
>> I tend to fall into the "release early, release often" camp in
>general,
>> but for this one I'm particularly anxious to get the faster Python
>direct
>> runner out in the hands of TFT/TFX users (and in particular have an
>eye on
>> https://www.tensorflow.org/dev-summit/ which I think can be a healthy
>> source of Beam users).
>>
>>
>> On Wed, Feb 28, 2018 at 7:01 PM Jean-Baptiste Onofré
>
>> wrote:
>>
>>> Hi Gus,
>>>
>>> Thanks for the update, it makes sense.
>>>
>>> Regards
>>> JB
>>>
>>> On 03/01/2018 02:59 AM, Konstantinos Katsiapis wrote:
>>> > Hi Jean-Baptiste,
>>> >
>>> > I can speak from the perspective of tf.transform
>>> >  (TFT) in particular and
>TFX
>>> >  libs in general,
>in
>>> case it is
>>> > useful.
>>> >
>>> > TFX distributed computation has 2 "large" dependencies, namely
>>> TensorFlow and
>>> > Apache Beam, each on their own release schedule.
>>> > As such, releasing of new TFX functionality often (but not always)
>>> depends on
>>> > (and is blocked by) releases of *both* TensorFlow *and* Apache
>Beam.
>>> >
>>> > Synchronizing releases across such large projects and
>organizations is
>>> likely
>>> > hard, so from our perspective having *frequent* releases of
>Tensorflow
>>> or Apache
>>> > Beam (and better yet both) decreases the time for which we are
>blocked
>>> on
>>> > releasing our features.
>>> >
>>> > In light of this, I would vote for more frequent releases in
>general,
>>> and for a
>>> > Beam 2.4 release soon in particular (as TFT 0.6 depends on it).
>>> >
>>> > Thanks,
>>> > Gus
>>> >
>>> > On Tue, Feb 27, 2018 at 10:29 PM, Jean-Baptiste Onofré
>>> > > wrote:
>>> >
>>> > By the way, if third party projects based on Beam (Google
>Dataflow,
>>> Talend
>>> > DataStream, and others) need a release (including some
>features),
>>> it's better to
>>> >  clearly state this on the mailing list.
>>> >
>>> > At Apache Karaf, I have lot of projects based on it
>(OpenDaylight,
>>> OpenHAB,
>>> > Websphere,  ...). They just ask for the release schedule and
>they
>>> align with
>>> > these release. As a best effort, I'm always trying to move
>fast
>>> when a release
>>> > is asked.
>>> >
>>> > So, if 2.4.0 is required by third party, no problem to "ask
>for a
>>> release".
>>> >
>>> > Regards
>>> > JB
>>> >
>>> > On 02/28/2018 04:17 AM, Reuven Lax wrote:
>>> > > It's been six weeks since you proposed beam 2.3.0. so
>assuming
>>> the same time
>>> > > scale for this release, that's 1.5 months between releases.
>>> Slightly faster than
>>> > > 2 months, but not by much.
>>> > >
>>> > > I do seem to remember that the original goal for beam was
>monthly
>>> releases though.
>>> > >
>>> > > Reuven
>>> > >
>>> > > On Tue, Feb 27, 2018, 9:12 PM Jean-Baptiste Onofré <
>>> j...@nanthrax.net 
>>> > > >> wrote:
>>> > >
>>> > > Hi Reuven,
>>> > >
>>> > > In a previous thread (about Beam project execution), I
>>> proposed a release every
>>> > > two months (as a best effort), I will find the e-mail.
>>> > >
>>> > > Beam 2.3.0 has been released "officially" on February
>16th,
>>> so two week ago
>>> > > roughly. I would have expected 2.4.0 not before end of
>March.
>>> > >
>>> > > If we have issue we want to fix fast, then 2.3.1 is
>good. If
>>> it's a new release
>>> > > in the pace, it's pretty fast and might "confuse" our
>users.
>>> > >
>>> > > That's why I'm curious ;)
>>> > >
>>> > > Regards
>>> > > JB
>>> > >
>>> > > On 02/28/2018 03:50 AM, Reuven Lax wrote:
>>> > >   

Re: Beam 2.4.0

[Robert Bradshaw]

Looking at the burn-down list, we have 5 remaining issues. None of these
are blockers, but all look like they're really close (reviewed, review
comments were addressed, waiting for a final LGTM). Specifically:

BEAM-3409 (teardown issues): Thomas Groh had some concerns, could you
verify they have all been addressed?
BEAM-3479 (DoFn classloader regression test): Kenn Knowles had minor
comments, looks like they were addressed, could you confirm?
BEAM-3735 (Missing gaming release archetypes): Lukasz Cwik had minor
comments, looks like they were addressed, could you confirm?
BEAM-3611 (KafkaIO.java splitting): Looks like this was resolved.
BEAM-3762 (unlimited JCE for Dataflow Worker): LGTM pending (currently
running) tests passing.

Let's see how many of these we can get in by, say, noon PST tomorrow.





On Wed, Feb 28, 2018 at 9:26 PM Robert Bradshaw  wrote:

> I tend to fall into the "release early, release often" camp in general,
> but for this one I'm particularly anxious to get the faster Python direct
> runner out in the hands of TFT/TFX users (and in particular have an eye on
> https://www.tensorflow.org/dev-summit/ which I think can be a healthy
> source of Beam users).
>
>
> On Wed, Feb 28, 2018 at 7:01 PM Jean-Baptiste Onofré 
> wrote:
>
>> Hi Gus,
>>
>> Thanks for the update, it makes sense.
>>
>> Regards
>> JB
>>
>> On 03/01/2018 02:59 AM, Konstantinos Katsiapis wrote:
>> > Hi Jean-Baptiste,
>> >
>> > I can speak from the perspective of tf.transform
>> >  (TFT) in particular and TFX
>> >  libs in general, in
>> case it is
>> > useful.
>> >
>> > TFX distributed computation has 2 "large" dependencies, namely
>> TensorFlow and
>> > Apache Beam, each on their own release schedule.
>> > As such, releasing of new TFX functionality often (but not always)
>> depends on
>> > (and is blocked by) releases of *both* TensorFlow *and* Apache Beam.
>> >
>> > Synchronizing releases across such large projects and organizations is
>> likely
>> > hard, so from our perspective having *frequent* releases of Tensorflow
>> or Apache
>> > Beam (and better yet both) decreases the time for which we are blocked
>> on
>> > releasing our features.
>> >
>> > In light of this, I would vote for more frequent releases in general,
>> and for a
>> > Beam 2.4 release soon in particular (as TFT 0.6 depends on it).
>> >
>> > Thanks,
>> > Gus
>> >
>> > On Tue, Feb 27, 2018 at 10:29 PM, Jean-Baptiste Onofré > > > wrote:
>> >
>> > By the way, if third party projects based on Beam (Google Dataflow,
>> Talend
>> > DataStream, and others) need a release (including some features),
>> it's better to
>> >  clearly state this on the mailing list.
>> >
>> > At Apache Karaf, I have lot of projects based on it (OpenDaylight,
>> OpenHAB,
>> > Websphere,  ...). They just ask for the release schedule and they
>> align with
>> > these release. As a best effort, I'm always trying to move fast
>> when a release
>> > is asked.
>> >
>> > So, if 2.4.0 is required by third party, no problem to "ask for a
>> release".
>> >
>> > Regards
>> > JB
>> >
>> > On 02/28/2018 04:17 AM, Reuven Lax wrote:
>> > > It's been six weeks since you proposed beam 2.3.0. so assuming
>> the same time
>> > > scale for this release, that's 1.5 months between releases.
>> Slightly faster than
>> > > 2 months, but not by much.
>> > >
>> > > I do seem to remember that the original goal for beam was monthly
>> releases though.
>> > >
>> > > Reuven
>> > >
>> > > On Tue, Feb 27, 2018, 9:12 PM Jean-Baptiste Onofré <
>> j...@nanthrax.net 
>> > > >> wrote:
>> > >
>> > > Hi Reuven,
>> > >
>> > > In a previous thread (about Beam project execution), I
>> proposed a release every
>> > > two months (as a best effort), I will find the e-mail.
>> > >
>> > > Beam 2.3.0 has been released "officially" on February 16th,
>> so two week ago
>> > > roughly. I would have expected 2.4.0 not before end of March.
>> > >
>> > > If we have issue we want to fix fast, then 2.3.1 is good. If
>> it's a new release
>> > > in the pace, it's pretty fast and might "confuse" our users.
>> > >
>> > > That's why I'm curious ;)
>> > >
>> > > Regards
>> > > JB
>> > >
>> > > On 02/28/2018 03:50 AM, Reuven Lax wrote:
>> > > > Wasn't the original statement monthly releases? We've never
>> realistically
>> > > > managed that, but Robert's proposed cut will be on a 6-week
>> pace.
>> > > >
>> > > > On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste Onofré <
>> j...@nanthrax.net 
>> > > 

Re: Beam 2.4.0

[Robert Bradshaw]

I tend to fall into the "release early, release often" camp in general, but
for this one I'm particularly anxious to get the faster Python direct
runner out in the hands of TFT/TFX users (and in particular have an eye on
https://www.tensorflow.org/dev-summit/ which I think can be a healthy
source of Beam users).


On Wed, Feb 28, 2018 at 7:01 PM Jean-Baptiste Onofré 
wrote:

> Hi Gus,
>
> Thanks for the update, it makes sense.
>
> Regards
> JB
>
> On 03/01/2018 02:59 AM, Konstantinos Katsiapis wrote:
> > Hi Jean-Baptiste,
> >
> > I can speak from the perspective of tf.transform
> >  (TFT) in particular and TFX
> >  libs in general, in
> case it is
> > useful.
> >
> > TFX distributed computation has 2 "large" dependencies, namely
> TensorFlow and
> > Apache Beam, each on their own release schedule.
> > As such, releasing of new TFX functionality often (but not always)
> depends on
> > (and is blocked by) releases of *both* TensorFlow *and* Apache Beam.
> >
> > Synchronizing releases across such large projects and organizations is
> likely
> > hard, so from our perspective having *frequent* releases of Tensorflow
> or Apache
> > Beam (and better yet both) decreases the time for which we are blocked on
> > releasing our features.
> >
> > In light of this, I would vote for more frequent releases in general,
> and for a
> > Beam 2.4 release soon in particular (as TFT 0.6 depends on it).
> >
> > Thanks,
> > Gus
> >
> > On Tue, Feb 27, 2018 at 10:29 PM, Jean-Baptiste Onofré  > > wrote:
> >
> > By the way, if third party projects based on Beam (Google Dataflow,
> Talend
> > DataStream, and others) need a release (including some features),
> it's better to
> >  clearly state this on the mailing list.
> >
> > At Apache Karaf, I have lot of projects based on it (OpenDaylight,
> OpenHAB,
> > Websphere,  ...). They just ask for the release schedule and they
> align with
> > these release. As a best effort, I'm always trying to move fast when
> a release
> > is asked.
> >
> > So, if 2.4.0 is required by third party, no problem to "ask for a
> release".
> >
> > Regards
> > JB
> >
> > On 02/28/2018 04:17 AM, Reuven Lax wrote:
> > > It's been six weeks since you proposed beam 2.3.0. so assuming the
> same time
> > > scale for this release, that's 1.5 months between releases.
> Slightly faster than
> > > 2 months, but not by much.
> > >
> > > I do seem to remember that the original goal for beam was monthly
> releases though.
> > >
> > > Reuven
> > >
> > > On Tue, Feb 27, 2018, 9:12 PM Jean-Baptiste Onofré <
> j...@nanthrax.net 
> > > >> wrote:
> > >
> > > Hi Reuven,
> > >
> > > In a previous thread (about Beam project execution), I
> proposed a release every
> > > two months (as a best effort), I will find the e-mail.
> > >
> > > Beam 2.3.0 has been released "officially" on February 16th, so
> two week ago
> > > roughly. I would have expected 2.4.0 not before end of March.
> > >
> > > If we have issue we want to fix fast, then 2.3.1 is good. If
> it's a new release
> > > in the pace, it's pretty fast and might "confuse" our users.
> > >
> > > That's why I'm curious ;)
> > >
> > > Regards
> > > JB
> > >
> > > On 02/28/2018 03:50 AM, Reuven Lax wrote:
> > > > Wasn't the original statement monthly releases? We've never
> realistically
> > > > managed that, but Robert's proposed cut will be on a 6-week
> pace.
> > > >
> > > > On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste Onofré <
> j...@nanthrax.net 
> > > >
> > > >   j...@nanthrax.net
> >  > > >
> > > > Hi Robert,
> > > >
> > > > I'm just curious: it's pretty fast compared to the
> original plan of a
> > > release
> > > > every two months. What's the reason to cut 2.4.0 now
> instead of end of
> > > March ?
> > > >
> > > > I will do the Jira triage and update today.
> > > >
> > > > Regards
> > > > JB
> > > >
> > > > On 02/27/2018 09:21 PM, Robert Bradshaw wrote:
> > > > > I'm planning on cutting the 2.4.0 release branch soon
> (tomorrow?). I
> > > see 13
> > > > > open issues on JIRA [1], none of which are labeled as
> blockers. If there
> > > > > are any that cannot be bumped to the next release, let
> me know soon.
> > > > >
> > > >

Re: Beam 2.4.0

[Jean-Baptiste Onofré]

Hi Gus,

Thanks for the update, it makes sense.

Regards
JB

On 03/01/2018 02:59 AM, Konstantinos Katsiapis wrote:
> Hi Jean-Baptiste,
> 
> I can speak from the perspective of tf.transform
>  (TFT) in particular and TFX
>  libs in general, in case it 
> is
> useful.
> 
> TFX distributed computation has 2 "large" dependencies, namely TensorFlow and
> Apache Beam, each on their own release schedule.
> As such, releasing of new TFX functionality often (but not always) depends on
> (and is blocked by) releases of *both* TensorFlow *and* Apache Beam.
> 
> Synchronizing releases across such large projects and organizations is likely
> hard, so from our perspective having *frequent* releases of Tensorflow or 
> Apache
> Beam (and better yet both) decreases the time for which we are blocked on
> releasing our features.
> 
> In light of this, I would vote for more frequent releases in general, and for 
> a
> Beam 2.4 release soon in particular (as TFT 0.6 depends on it).
> 
> Thanks,
> Gus
> 
> On Tue, Feb 27, 2018 at 10:29 PM, Jean-Baptiste Onofré  > wrote:
> 
> By the way, if third party projects based on Beam (Google Dataflow, Talend
> DataStream, and others) need a release (including some features), it's 
> better to
>  clearly state this on the mailing list.
> 
> At Apache Karaf, I have lot of projects based on it (OpenDaylight, 
> OpenHAB,
> Websphere,  ...). They just ask for the release schedule and they align 
> with
> these release. As a best effort, I'm always trying to move fast when a 
> release
> is asked.
> 
> So, if 2.4.0 is required by third party, no problem to "ask for a 
> release".
> 
> Regards
> JB
> 
> On 02/28/2018 04:17 AM, Reuven Lax wrote:
> > It's been six weeks since you proposed beam 2.3.0. so assuming the same 
> time
> > scale for this release, that's 1.5 months between releases. Slightly 
> faster than
> > 2 months, but not by much.
> >
> > I do seem to remember that the original goal for beam was monthly 
> releases though.
> >
> > Reuven
> >
> > On Tue, Feb 27, 2018, 9:12 PM Jean-Baptiste Onofré  
> > >> wrote:
> >
> >     Hi Reuven,
> >
> >     In a previous thread (about Beam project execution), I proposed a 
> release every
> >     two months (as a best effort), I will find the e-mail.
> >
> >     Beam 2.3.0 has been released "officially" on February 16th, so two 
> week ago
> >     roughly. I would have expected 2.4.0 not before end of March.
> >
> >     If we have issue we want to fix fast, then 2.3.1 is good. If it's a 
> new release
> >     in the pace, it's pretty fast and might "confuse" our users.
> >
> >     That's why I'm curious ;)
> >
> >     Regards
> >     JB
> >
> >     On 02/28/2018 03:50 AM, Reuven Lax wrote:
> >     > Wasn't the original statement monthly releases? We've never 
> realistically
> >     > managed that, but Robert's proposed cut will be on a 6-week pace.
> >     >
> >     > On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste Onofré 
> 
> >     >
> >     >  
>   >     >
> >     >     Hi Robert,
> >     >
> >     >     I'm just curious: it's pretty fast compared to the original 
> plan of a
> >     release
> >     >     every two months. What's the reason to cut 2.4.0 now instead 
> of end of
> >     March ?
> >     >
> >     >     I will do the Jira triage and update today.
> >     >
> >     >     Regards
> >     >     JB
> >     >
> >     >     On 02/27/2018 09:21 PM, Robert Bradshaw wrote:
> >     >     > I'm planning on cutting the 2.4.0 release branch soon 
> (tomorrow?). I
> >     see 13
> >     >     > open issues on JIRA [1], none of which are labeled as 
> blockers. If there
> >     >     > are any that cannot be bumped to the next release, let me 
> know soon.
> >     >     >
> >     >     > - Robert
> >     >     >
> >     >     >
> >     >     > [1]
> >     >     >
> >     >   
> >      
> https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
> 
> 
> >     >     >
> >     >
> >     >     --
> >     >     

Re: Beam 2.4.0

[Konstantinos Katsiapis]

Hi Jean-Baptiste,

I can speak from the perspective of tf.transform
 (TFT) in particular and TFX
 libs in general, in case
it is useful.

TFX distributed computation has 2 "large" dependencies, namely TensorFlow
and Apache Beam, each on their own release schedule.
As such, releasing of new TFX functionality often (but not always) depends
on (and is blocked by) releases of *both* TensorFlow *and* Apache Beam.

Synchronizing releases across such large projects and organizations is
likely hard, so from our perspective having *frequent* releases of
Tensorflow or Apache Beam (and better yet both) decreases the time for
which we are blocked on releasing our features.

In light of this, I would vote for more frequent releases in general, and
for a Beam 2.4 release soon in particular (as TFT 0.6 depends on it).

Thanks,
Gus

On Tue, Feb 27, 2018 at 10:29 PM, Jean-Baptiste Onofré 
wrote:

> By the way, if third party projects based on Beam (Google Dataflow, Talend
> DataStream, and others) need a release (including some features), it's
> better to
>  clearly state this on the mailing list.
>
> At Apache Karaf, I have lot of projects based on it (OpenDaylight, OpenHAB,
> Websphere,  ...). They just ask for the release schedule and they align
> with
> these release. As a best effort, I'm always trying to move fast when a
> release
> is asked.
>
> So, if 2.4.0 is required by third party, no problem to "ask for a release".
>
> Regards
> JB
>
> On 02/28/2018 04:17 AM, Reuven Lax wrote:
> > It's been six weeks since you proposed beam 2.3.0. so assuming the same
> time
> > scale for this release, that's 1.5 months between releases. Slightly
> faster than
> > 2 months, but not by much.
> >
> > I do seem to remember that the original goal for beam was monthly
> releases though.
> >
> > Reuven
> >
> > On Tue, Feb 27, 2018, 9:12 PM Jean-Baptiste Onofré  > > wrote:
> >
> > Hi Reuven,
> >
> > In a previous thread (about Beam project execution), I proposed a
> release every
> > two months (as a best effort), I will find the e-mail.
> >
> > Beam 2.3.0 has been released "officially" on February 16th, so two
> week ago
> > roughly. I would have expected 2.4.0 not before end of March.
> >
> > If we have issue we want to fix fast, then 2.3.1 is good. If it's a
> new release
> > in the pace, it's pretty fast and might "confuse" our users.
> >
> > That's why I'm curious ;)
> >
> > Regards
> > JB
> >
> > On 02/28/2018 03:50 AM, Reuven Lax wrote:
> > > Wasn't the original statement monthly releases? We've never
> realistically
> > > managed that, but Robert's proposed cut will be on a 6-week pace.
> > >
> > > On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste Onofré <
> j...@nanthrax.net
> > 
> > > >> wrote:
> > >
> > > Hi Robert,
> > >
> > > I'm just curious: it's pretty fast compared to the original
> plan of a
> > release
> > > every two months. What's the reason to cut 2.4.0 now instead
> of end of
> > March ?
> > >
> > > I will do the Jira triage and update today.
> > >
> > > Regards
> > > JB
> > >
> > > On 02/27/2018 09:21 PM, Robert Bradshaw wrote:
> > > > I'm planning on cutting the 2.4.0 release branch soon
> (tomorrow?). I
> > see 13
> > > > open issues on JIRA [1], none of which are labeled as
> blockers. If there
> > > > are any that cannot be bumped to the next release, let me
> know soon.
> > > >
> > > > - Robert
> > > >
> > > >
> > > > [1]
> > > >
> > >
> >  https://issues.apache.org/jira/browse/BEAM-3749?jql=
> project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%
> 22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
> > > >
> > >
> > > --
> > > Jean-Baptiste Onofré
> > > jbono...@apache.org 
> > >
> > > http://blog.nanthrax.net
> > > Talend - http://www.talend.com
> > >
> >
> > --
> > Jean-Baptiste Onofré
> > jbono...@apache.org 
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>



-- 
Gus Katsiapis | Software Engineer | katsia...@google.com | 650-918-7487

Re: Beam 2.4.0

[Jean-Baptiste Onofré]

By the way, if third party projects based on Beam (Google Dataflow, Talend
DataStream, and others) need a release (including some features), it's better to
 clearly state this on the mailing list.

At Apache Karaf, I have lot of projects based on it (OpenDaylight, OpenHAB,
Websphere,  ...). They just ask for the release schedule and they align with
these release. As a best effort, I'm always trying to move fast when a release
is asked.

So, if 2.4.0 is required by third party, no problem to "ask for a release".

Regards
JB

On 02/28/2018 04:17 AM, Reuven Lax wrote:
> It's been six weeks since you proposed beam 2.3.0. so assuming the same time
> scale for this release, that's 1.5 months between releases. Slightly faster 
> than
> 2 months, but not by much.
> 
> I do seem to remember that the original goal for beam was monthly releases 
> though.
> 
> Reuven
> 
> On Tue, Feb 27, 2018, 9:12 PM Jean-Baptiste Onofré  > wrote:
> 
> Hi Reuven,
> 
> In a previous thread (about Beam project execution), I proposed a release 
> every
> two months (as a best effort), I will find the e-mail.
> 
> Beam 2.3.0 has been released "officially" on February 16th, so two week 
> ago
> roughly. I would have expected 2.4.0 not before end of March.
> 
> If we have issue we want to fix fast, then 2.3.1 is good. If it's a new 
> release
> in the pace, it's pretty fast and might "confuse" our users.
> 
> That's why I'm curious ;)
> 
> Regards
> JB
> 
> On 02/28/2018 03:50 AM, Reuven Lax wrote:
> > Wasn't the original statement monthly releases? We've never 
> realistically
> > managed that, but Robert's proposed cut will be on a 6-week pace.
> >
> > On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste Onofré  
> > >> wrote:
> >
> >     Hi Robert,
> >
> >     I'm just curious: it's pretty fast compared to the original plan of 
> a
> release
> >     every two months. What's the reason to cut 2.4.0 now instead of end 
> of
> March ?
> >
> >     I will do the Jira triage and update today.
> >
> >     Regards
> >     JB
> >
> >     On 02/27/2018 09:21 PM, Robert Bradshaw wrote:
> >     > I'm planning on cutting the 2.4.0 release branch soon 
> (tomorrow?). I
> see 13
> >     > open issues on JIRA [1], none of which are labeled as blockers. 
> If there
> >     > are any that cannot be bumped to the next release, let me know 
> soon.
> >     >
> >     > - Robert
> >     >
> >     >
> >     > [1]
> >     >
> >   
>  
> https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
> >     >
> >
> >     --
> >     Jean-Baptiste Onofré
> >     jbono...@apache.org 
> >
> >     http://blog.nanthrax.net
> >     Talend - http://www.talend.com
> >
> 
> --
> Jean-Baptiste Onofré
> jbono...@apache.org 
> http://blog.nanthrax.net
> Talend - http://www.talend.com
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Beam 2.4.0

[Jean-Baptiste Onofré]

A month is OK for me: I prefer that than longer (as it was before).

The most important is a regular pace: we should avoid a 2.4.0 now and a 2.5.0
four months later. So, it means that 2.5.0 should happen roughly end of April.

I will send a clear statement (as I already did weeks ago) around releases pace
and schedule.

I'm volunteer to drive the releases.

Regards
JB

On 02/28/2018 06:43 AM, Kenneth Knowles wrote:
> To be semver, 2.3.1 should be rollback safe with 2.3.0. Normally that is
> accomplished by cutting 2.3.1 release branch from 2.3.0 release branch and 
> then
> have fixes cherrypicked.
> 
> I think 6 weeks between minor version releases is not too fast. I think a 
> month
> is a good target.
> 
> We tend to have high latency between cut and release, so we should plan on 
> that.
> 
> Kenn
> 
> On Tue, Feb 27, 2018 at 9:16 PM, Jean-Baptiste Onofré  > wrote:
> 
> That's the point of my e-mail indeed. I think it would make more sense for
> users.
> 
> Regards
> JB
> 
> On 02/28/2018 06:04 AM, Romain Manni-Bucau wrote:
> > Thinking out loud: why not trying a 2.3.1 with small fixes only and the 
> 2.4
> > after 6 weeks starting from the 2.3.0 real release date.
> >
> > Le 28 févr. 2018 04:24, "Jean-Baptiste Onofré"  
> > >> a écrit :
> >
> >     OK, maybe I wasn't clear: for me the cycle is ~ 6 weeks once a 
> release is out,
> >     not when it started.
> >
> >     I don't remember about monthly release (it's too fast IMHO).
> >
> >     Anyway, let me find the thread dealing with release pace and 
> propose a clear
> >     statement. It's important for our users.
> >
> >     Regards
> >     JB
> >
> >     On 02/28/2018 04:17 AM, Reuven Lax wrote:
> >     > It's been six weeks since you proposed beam 2.3.0. so assuming 
> the same time
> >     > scale for this release, that's 1.5 months between releases. 
> Slightly
> >     faster than
> >     > 2 months, but not by much.
> >     >
> >     > I do seem to remember that the original goal for beam was monthly 
> releases
> >     though.
> >     >
> >     > Reuven
> >     >
> >     > On Tue, Feb 27, 2018, 9:12 PM Jean-Baptiste Onofré 
> 
> >     >
> >     > 
>  >     >
> >     >     Hi Reuven,
> >     >
> >     >     In a previous thread (about Beam project execution), I 
> proposed a
> >     release every
> >     >     two months (as a best effort), I will find the e-mail.
> >     >
> >     >     Beam 2.3.0 has been released "officially" on February 16th, 
> so two
> >     week ago
> >     >     roughly. I would have expected 2.4.0 not before end of March.
> >     >
> >     >     If we have issue we want to fix fast, then 2.3.1 is good. If 
> it's a
> >     new release
> >     >     in the pace, it's pretty fast and might "confuse" our users.
> >     >
> >     >     That's why I'm curious ;)
> >     >
> >     >     Regards
> >     >     JB
> >     >
> >     >     On 02/28/2018 03:50 AM, Reuven Lax wrote:
> >     >     > Wasn't the original statement monthly releases? We've never
> >     realistically
> >     >     > managed that, but Robert's proposed cut will be on a 6-week 
> pace.
> >     >     >
> >     >     > On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste Onofré
> 
> >     >
> >     >     
> >>
> >     >     > 
> >
> >     
>  wrote:
> >     >     >
> >     >     >     Hi Robert,
> >     >     >
> >     >     >     I'm just curious: it's pretty fast compared to the
> original plan
> >     of a
> >     >     release
> >     >     >     every two months. What's the reason to cut 2.4.0 now
> instead of
> >     end of
> >     >     March ?
> >     >     >
> >     >     >     I will do the Jira triage and update today.
> >     >     >
> >     >     >     Regards
> >     >     >     JB
> >     >     >
> >     >     >     On 02/27/2018 09:21 PM, Robert Bradshaw wrote:
> >     >     >     > I'm planning on cutting the 2.4.0 release branch soon
>  

Re: Beam 2.4.0

[Kenneth Knowles]

To be semver, 2.3.1 should be rollback safe with 2.3.0. Normally that is
accomplished by cutting 2.3.1 release branch from 2.3.0 release branch and
then have fixes cherrypicked.

I think 6 weeks between minor version releases is not too fast. I think a
month is a good target.

We tend to have high latency between cut and release, so we should plan on
that.

Kenn

On Tue, Feb 27, 2018 at 9:16 PM, Jean-Baptiste Onofré 
wrote:

> That's the point of my e-mail indeed. I think it would make more sense for
> users.
>
> Regards
> JB
>
> On 02/28/2018 06:04 AM, Romain Manni-Bucau wrote:
> > Thinking out loud: why not trying a 2.3.1 with small fixes only and the
> 2.4
> > after 6 weeks starting from the 2.3.0 real release date.
> >
> > Le 28 févr. 2018 04:24, "Jean-Baptiste Onofré"  > > a écrit :
> >
> > OK, maybe I wasn't clear: for me the cycle is ~ 6 weeks once a
> release is out,
> > not when it started.
> >
> > I don't remember about monthly release (it's too fast IMHO).
> >
> > Anyway, let me find the thread dealing with release pace and propose
> a clear
> > statement. It's important for our users.
> >
> > Regards
> > JB
> >
> > On 02/28/2018 04:17 AM, Reuven Lax wrote:
> > > It's been six weeks since you proposed beam 2.3.0. so assuming the
> same time
> > > scale for this release, that's 1.5 months between releases.
> Slightly
> > faster than
> > > 2 months, but not by much.
> > >
> > > I do seem to remember that the original goal for beam was monthly
> releases
> > though.
> > >
> > > Reuven
> > >
> > > On Tue, Feb 27, 2018, 9:12 PM Jean-Baptiste Onofré <
> j...@nanthrax.net
> > 
> > > >> wrote:
> > >
> > > Hi Reuven,
> > >
> > > In a previous thread (about Beam project execution), I
> proposed a
> > release every
> > > two months (as a best effort), I will find the e-mail.
> > >
> > > Beam 2.3.0 has been released "officially" on February 16th, so
> two
> > week ago
> > > roughly. I would have expected 2.4.0 not before end of March.
> > >
> > > If we have issue we want to fix fast, then 2.3.1 is good. If
> it's a
> > new release
> > > in the pace, it's pretty fast and might "confuse" our users.
> > >
> > > That's why I'm curious ;)
> > >
> > > Regards
> > > JB
> > >
> > > On 02/28/2018 03:50 AM, Reuven Lax wrote:
> > > > Wasn't the original statement monthly releases? We've never
> > realistically
> > > > managed that, but Robert's proposed cut will be on a 6-week
> pace.
> > > >
> > > > On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste Onofré <
> j...@nanthrax.net
> > 
> > > >
> > > > 
> >  > > >
> > > > Hi Robert,
> > > >
> > > > I'm just curious: it's pretty fast compared to the
> original plan
> > of a
> > > release
> > > > every two months. What's the reason to cut 2.4.0 now
> instead of
> > end of
> > > March ?
> > > >
> > > > I will do the Jira triage and update today.
> > > >
> > > > Regards
> > > > JB
> > > >
> > > > On 02/27/2018 09:21 PM, Robert Bradshaw wrote:
> > > > > I'm planning on cutting the 2.4.0 release branch soon
> > (tomorrow?). I
> > > see 13
> > > > > open issues on JIRA [1], none of which are labeled as
> > blockers. If there
> > > > > are any that cannot be bumped to the next release, let
> me know
> > soon.
> > > > >
> > > > > - Robert
> > > > >
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >   https://issues.apache.org/jira/browse/BEAM-3749?jql=
> project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%
> 22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
> >  project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%
> 22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0>
> > > > >
> > > >
> > > > --
> > > > Jean-Baptiste Onofré
> > > > jbono...@apache.org 
> > >
> > > 
> > >>
> > > > http://blog.nanthrax.net
> > > > Talend 

Re: Beam 2.4.0

[Jean-Baptiste Onofré]

That's the point of my e-mail indeed. I think it would make more sense for 
users.

Regards
JB

On 02/28/2018 06:04 AM, Romain Manni-Bucau wrote:
> Thinking out loud: why not trying a 2.3.1 with small fixes only and the 2.4
> after 6 weeks starting from the 2.3.0 real release date.
> 
> Le 28 févr. 2018 04:24, "Jean-Baptiste Onofré"  > a écrit :
> 
> OK, maybe I wasn't clear: for me the cycle is ~ 6 weeks once a release is 
> out,
> not when it started.
> 
> I don't remember about monthly release (it's too fast IMHO).
> 
> Anyway, let me find the thread dealing with release pace and propose a 
> clear
> statement. It's important for our users.
> 
> Regards
> JB
> 
> On 02/28/2018 04:17 AM, Reuven Lax wrote:
> > It's been six weeks since you proposed beam 2.3.0. so assuming the same 
> time
> > scale for this release, that's 1.5 months between releases. Slightly
> faster than
> > 2 months, but not by much.
> >
> > I do seem to remember that the original goal for beam was monthly 
> releases
> though.
> >
> > Reuven
> >
> > On Tue, Feb 27, 2018, 9:12 PM Jean-Baptiste Onofré  
> > >> wrote:
> >
> >     Hi Reuven,
> >
> >     In a previous thread (about Beam project execution), I proposed a
> release every
> >     two months (as a best effort), I will find the e-mail.
> >
> >     Beam 2.3.0 has been released "officially" on February 16th, so two
> week ago
> >     roughly. I would have expected 2.4.0 not before end of March.
> >
> >     If we have issue we want to fix fast, then 2.3.1 is good. If it's a
> new release
> >     in the pace, it's pretty fast and might "confuse" our users.
> >
> >     That's why I'm curious ;)
> >
> >     Regards
> >     JB
> >
> >     On 02/28/2018 03:50 AM, Reuven Lax wrote:
> >     > Wasn't the original statement monthly releases? We've never
> realistically
> >     > managed that, but Robert's proposed cut will be on a 6-week pace.
> >     >
> >     > On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste Onofré 
>  
> >     >
> >     > 
>  >     >
> >     >     Hi Robert,
> >     >
> >     >     I'm just curious: it's pretty fast compared to the original 
> plan
> of a
> >     release
> >     >     every two months. What's the reason to cut 2.4.0 now instead 
> of
> end of
> >     March ?
> >     >
> >     >     I will do the Jira triage and update today.
> >     >
> >     >     Regards
> >     >     JB
> >     >
> >     >     On 02/27/2018 09:21 PM, Robert Bradshaw wrote:
> >     >     > I'm planning on cutting the 2.4.0 release branch soon
> (tomorrow?). I
> >     see 13
> >     >     > open issues on JIRA [1], none of which are labeled as
> blockers. If there
> >     >     > are any that cannot be bumped to the next release, let me 
> know
> soon.
> >     >     >
> >     >     > - Robert
> >     >     >
> >     >     >
> >     >     > [1]
> >     >     >
> >     >   
> >   
>   
> https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
> 
> 
> >     >     >
> >     >
> >     >     --
> >     >     Jean-Baptiste Onofré
> >     >     jbono...@apache.org 
> >
> >     
> >>
> >     >     http://blog.nanthrax.net
> >     >     Talend - http://www.talend.com
> >     >
> >
> >     --
> >     Jean-Baptiste Onofré
> >     jbono...@apache.org 
> >
> >     http://blog.nanthrax.net
> >     Talend - http://www.talend.com
> >
> 
> --
> Jean-Baptiste Onofré
> jbono...@apache.org 
> http://blog.nanthrax.net
> Talend - http://www.talend.com
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Beam 2.4.0

[Romain Manni-Bucau]

Thinking out loud: why not trying a 2.3.1 with small fixes only and the 2.4
after 6 weeks starting from the 2.3.0 real release date.

Le 28 févr. 2018 04:24, "Jean-Baptiste Onofré"  a écrit :

> OK, maybe I wasn't clear: for me the cycle is ~ 6 weeks once a release is
> out,
> not when it started.
>
> I don't remember about monthly release (it's too fast IMHO).
>
> Anyway, let me find the thread dealing with release pace and propose a
> clear
> statement. It's important for our users.
>
> Regards
> JB
>
> On 02/28/2018 04:17 AM, Reuven Lax wrote:
> > It's been six weeks since you proposed beam 2.3.0. so assuming the same
> time
> > scale for this release, that's 1.5 months between releases. Slightly
> faster than
> > 2 months, but not by much.
> >
> > I do seem to remember that the original goal for beam was monthly
> releases though.
> >
> > Reuven
> >
> > On Tue, Feb 27, 2018, 9:12 PM Jean-Baptiste Onofré  > > wrote:
> >
> > Hi Reuven,
> >
> > In a previous thread (about Beam project execution), I proposed a
> release every
> > two months (as a best effort), I will find the e-mail.
> >
> > Beam 2.3.0 has been released "officially" on February 16th, so two
> week ago
> > roughly. I would have expected 2.4.0 not before end of March.
> >
> > If we have issue we want to fix fast, then 2.3.1 is good. If it's a
> new release
> > in the pace, it's pretty fast and might "confuse" our users.
> >
> > That's why I'm curious ;)
> >
> > Regards
> > JB
> >
> > On 02/28/2018 03:50 AM, Reuven Lax wrote:
> > > Wasn't the original statement monthly releases? We've never
> realistically
> > > managed that, but Robert's proposed cut will be on a 6-week pace.
> > >
> > > On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste Onofré <
> j...@nanthrax.net
> > 
> > > >> wrote:
> > >
> > > Hi Robert,
> > >
> > > I'm just curious: it's pretty fast compared to the original
> plan of a
> > release
> > > every two months. What's the reason to cut 2.4.0 now instead
> of end of
> > March ?
> > >
> > > I will do the Jira triage and update today.
> > >
> > > Regards
> > > JB
> > >
> > > On 02/27/2018 09:21 PM, Robert Bradshaw wrote:
> > > > I'm planning on cutting the 2.4.0 release branch soon
> (tomorrow?). I
> > see 13
> > > > open issues on JIRA [1], none of which are labeled as
> blockers. If there
> > > > are any that cannot be bumped to the next release, let me
> know soon.
> > > >
> > > > - Robert
> > > >
> > > >
> > > > [1]
> > > >
> > >
> >  https://issues.apache.org/jira/browse/BEAM-3749?jql=
> project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%
> 22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
> > > >
> > >
> > > --
> > > Jean-Baptiste Onofré
> > > jbono...@apache.org 
> > >
> > > http://blog.nanthrax.net
> > > Talend - http://www.talend.com
> > >
> >
> > --
> > Jean-Baptiste Onofré
> > jbono...@apache.org 
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: Beam 2.4.0

[Jean-Baptiste Onofré]

OK, maybe I wasn't clear: for me the cycle is ~ 6 weeks once a release is out,
not when it started.

I don't remember about monthly release (it's too fast IMHO).

Anyway, let me find the thread dealing with release pace and propose a clear
statement. It's important for our users.

Regards
JB

On 02/28/2018 04:17 AM, Reuven Lax wrote:
> It's been six weeks since you proposed beam 2.3.0. so assuming the same time
> scale for this release, that's 1.5 months between releases. Slightly faster 
> than
> 2 months, but not by much.
> 
> I do seem to remember that the original goal for beam was monthly releases 
> though.
> 
> Reuven
> 
> On Tue, Feb 27, 2018, 9:12 PM Jean-Baptiste Onofré  > wrote:
> 
> Hi Reuven,
> 
> In a previous thread (about Beam project execution), I proposed a release 
> every
> two months (as a best effort), I will find the e-mail.
> 
> Beam 2.3.0 has been released "officially" on February 16th, so two week 
> ago
> roughly. I would have expected 2.4.0 not before end of March.
> 
> If we have issue we want to fix fast, then 2.3.1 is good. If it's a new 
> release
> in the pace, it's pretty fast and might "confuse" our users.
> 
> That's why I'm curious ;)
> 
> Regards
> JB
> 
> On 02/28/2018 03:50 AM, Reuven Lax wrote:
> > Wasn't the original statement monthly releases? We've never 
> realistically
> > managed that, but Robert's proposed cut will be on a 6-week pace.
> >
> > On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste Onofré  
> > >> wrote:
> >
> >     Hi Robert,
> >
> >     I'm just curious: it's pretty fast compared to the original plan of 
> a
> release
> >     every two months. What's the reason to cut 2.4.0 now instead of end 
> of
> March ?
> >
> >     I will do the Jira triage and update today.
> >
> >     Regards
> >     JB
> >
> >     On 02/27/2018 09:21 PM, Robert Bradshaw wrote:
> >     > I'm planning on cutting the 2.4.0 release branch soon 
> (tomorrow?). I
> see 13
> >     > open issues on JIRA [1], none of which are labeled as blockers. 
> If there
> >     > are any that cannot be bumped to the next release, let me know 
> soon.
> >     >
> >     > - Robert
> >     >
> >     >
> >     > [1]
> >     >
> >   
>  
> https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
> >     >
> >
> >     --
> >     Jean-Baptiste Onofré
> >     jbono...@apache.org 
> >
> >     http://blog.nanthrax.net
> >     Talend - http://www.talend.com
> >
> 
> --
> Jean-Baptiste Onofré
> jbono...@apache.org 
> http://blog.nanthrax.net
> Talend - http://www.talend.com
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Beam 2.4.0

[Reuven Lax]

It's been six weeks since you proposed beam 2.3.0. so assuming the same
time scale for this release, that's 1.5 months between releases. Slightly
faster than 2 months, but not by much.

I do seem to remember that the original goal for beam was monthly releases
though.

Reuven

On Tue, Feb 27, 2018, 9:12 PM Jean-Baptiste Onofré  wrote:

> Hi Reuven,
>
> In a previous thread (about Beam project execution), I proposed a release
> every
> two months (as a best effort), I will find the e-mail.
>
> Beam 2.3.0 has been released "officially" on February 16th, so two week ago
> roughly. I would have expected 2.4.0 not before end of March.
>
> If we have issue we want to fix fast, then 2.3.1 is good. If it's a new
> release
> in the pace, it's pretty fast and might "confuse" our users.
>
> That's why I'm curious ;)
>
> Regards
> JB
>
> On 02/28/2018 03:50 AM, Reuven Lax wrote:
> > Wasn't the original statement monthly releases? We've never realistically
> > managed that, but Robert's proposed cut will be on a 6-week pace.
> >
> > On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste Onofré  > > wrote:
> >
> > Hi Robert,
> >
> > I'm just curious: it's pretty fast compared to the original plan of
> a release
> > every two months. What's the reason to cut 2.4.0 now instead of end
> of March ?
> >
> > I will do the Jira triage and update today.
> >
> > Regards
> > JB
> >
> > On 02/27/2018 09:21 PM, Robert Bradshaw wrote:
> > > I'm planning on cutting the 2.4.0 release branch soon (tomorrow?).
> I see 13
> > > open issues on JIRA [1], none of which are labeled as blockers. If
> there
> > > are any that cannot be bumped to the next release, let me know
> soon.
> > >
> > > - Robert
> > >
> > >
> > > [1]
> > >
> >
> https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
> > >
> >
> > --
> > Jean-Baptiste Onofré
> > jbono...@apache.org 
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: Beam 2.4.0

[Jean-Baptiste Onofré]

Hi Reuven,

In a previous thread (about Beam project execution), I proposed a release every
two months (as a best effort), I will find the e-mail.

Beam 2.3.0 has been released "officially" on February 16th, so two week ago
roughly. I would have expected 2.4.0 not before end of March.

If we have issue we want to fix fast, then 2.3.1 is good. If it's a new release
in the pace, it's pretty fast and might "confuse" our users.

That's why I'm curious ;)

Regards
JB

On 02/28/2018 03:50 AM, Reuven Lax wrote:
> Wasn't the original statement monthly releases? We've never realistically
> managed that, but Robert's proposed cut will be on a 6-week pace.
> 
> On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste Onofré  > wrote:
> 
> Hi Robert,
> 
> I'm just curious: it's pretty fast compared to the original plan of a 
> release
> every two months. What's the reason to cut 2.4.0 now instead of end of 
> March ?
> 
> I will do the Jira triage and update today.
> 
> Regards
> JB
> 
> On 02/27/2018 09:21 PM, Robert Bradshaw wrote:
> > I'm planning on cutting the 2.4.0 release branch soon (tomorrow?). I 
> see 13
> > open issues on JIRA [1], none of which are labeled as blockers. If there
> > are any that cannot be bumped to the next release, let me know soon.
> >
> > - Robert
> >
> >
> > [1]
> >
> 
> https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
> >
> 
> --
> Jean-Baptiste Onofré
> jbono...@apache.org 
> http://blog.nanthrax.net
> Talend - http://www.talend.com
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Beam 2.4.0

[Reuven Lax]

Wasn't the original statement monthly releases? We've never realistically
managed that, but Robert's proposed cut will be on a 6-week pace.

On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste Onofré  wrote:

> Hi Robert,
>
> I'm just curious: it's pretty fast compared to the original plan of a
> release
> every two months. What's the reason to cut 2.4.0 now instead of end of
> March ?
>
> I will do the Jira triage and update today.
>
> Regards
> JB
>
> On 02/27/2018 09:21 PM, Robert Bradshaw wrote:
> > I'm planning on cutting the 2.4.0 release branch soon (tomorrow?). I see
> 13
> > open issues on JIRA [1], none of which are labeled as blockers. If there
> > are any that cannot be bumped to the next release, let me know soon.
> >
> > - Robert
> >
> >
> > [1]
> >
> https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
> >
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: Beam 2.4.0

[Jean-Baptiste Onofré]

Hi Robert,

I'm just curious: it's pretty fast compared to the original plan of a release
every two months. What's the reason to cut 2.4.0 now instead of end of March ?

I will do the Jira triage and update today.

Regards
JB

On 02/27/2018 09:21 PM, Robert Bradshaw wrote:
> I'm planning on cutting the 2.4.0 release branch soon (tomorrow?). I see 13
> open issues on JIRA [1], none of which are labeled as blockers. If there
> are any that cannot be bumped to the next release, let me know soon.
> 
> - Robert
> 
> 
> [1]
> https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Beam 2.4.0

[Reuven Lax]

The issue isn't that we miss teardown, it's that WaitUntilFinish doesn't
wait until all teardown calls have finished. However this is nearly as bad
as not calling teardown at all, and can result in flaky tests.

I think we should make an effort to get this fix into 2.4.0, but I also
don't think we should block 2.4.0 on it. The reason being is that this is a
long-standing issue (it's been a bug since Teardown was introduced) and
it's not a production data-correctness issue; in the past we've stated that
we will not block releases for bugs like this, however we will make an
attempt to include the bugfixes if possible.

Reuven


On Tue, Feb 27, 2018 at 1:42 PM Romain Manni-Bucau <rmannibu...@gmail.com>
wrote:

> Not sure when it appeared but running any test you miss the teardown which
> is:
>
> 1. A part you cant test today cause of that
> 2. Leaking and can impact other tests depending the impl and setup
> 3. Prevent any correct concurrency plannification of tests (even using the
> random of the direct runner you can plan it but not with teardown leaks
> leading to numerous unexpected threads)
>
> Can work for simple cases like the IO in beam but this week (since
> yesterday ;)) already 2 users reported me broken executions due to that and
> I was only able to say the PR was ready since weeks :(. Since code fix is
> here I think it should make it as a minimum to make the directrunner usable
> for tests. Cleanup of the code - dont think there is any - can come later
> if needed but normally all was addressed anyway and doesnt impact code
> quality.
>
> Le 27 févr. 2018 22:34, "Reuven Lax" <re...@google.com> a écrit :
>
>> Can you explain "unusable?" We have hundreds of direct-runner tests that
>> appear to still be running just fine. Is this a new regression, or old
>> behavior?
>>
>>
>> On Tue, Feb 27, 2018 at 1:30 PM Romain Manni-Bucau <rmannibu...@gmail.com>
>> wrote:
>>
>>> Updated 3409 as blocker since it makes direct runner - and therefore any
>>> test - unusable at all (leaks+unexpected wait times + retry strategy
>>> required+no real alternative even for dofn now dofntester is deprecated).
>>>
>>> Le 27 févr. 2018 21:45, "Chamikara Jayalath" <chamik...@google.com> a
>>> écrit :
>>>
>>>> Looks like previous URL was broken. Created
>>>> https://s.apache.org/beam-2.4.0-burndown.
>>>>
>>>> - Cham
>>>>
>>>> On Tue, Feb 27, 2018 at 12:22 PM Robert Bradshaw <rober...@google.com>
>>>> wrote:
>>>>
>>>>> I'm planning on cutting the 2.4.0 release branch soon (tomorrow?). I
>>>>> see 13
>>>>> open issues on JIRA [1], none of which are labeled as blockers. If
>>>>> there
>>>>> are any that cannot be bumped to the next release, let me know soon.
>>>>>
>>>>> - Robert
>>>>>
>>>>>
>>>>> [1]
>>>>>
>>>>> https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
>>>>>
>>>>

Re: Beam 2.4.0

[Romain Manni-Bucau]

Not sure when it appeared but running any test you miss the teardown which
is:

1. A part you cant test today cause of that
2. Leaking and can impact other tests depending the impl and setup
3. Prevent any correct concurrency plannification of tests (even using the
random of the direct runner you can plan it but not with teardown leaks
leading to numerous unexpected threads)

Can work for simple cases like the IO in beam but this week (since
yesterday ;)) already 2 users reported me broken executions due to that and
I was only able to say the PR was ready since weeks :(. Since code fix is
here I think it should make it as a minimum to make the directrunner usable
for tests. Cleanup of the code - dont think there is any - can come later
if needed but normally all was addressed anyway and doesnt impact code
quality.

Le 27 févr. 2018 22:34, "Reuven Lax" <re...@google.com> a écrit :

> Can you explain "unusable?" We have hundreds of direct-runner tests that
> appear to still be running just fine. Is this a new regression, or old
> behavior?
>
>
> On Tue, Feb 27, 2018 at 1:30 PM Romain Manni-Bucau <rmannibu...@gmail.com>
> wrote:
>
>> Updated 3409 as blocker since it makes direct runner - and therefore any
>> test - unusable at all (leaks+unexpected wait times + retry strategy
>> required+no real alternative even for dofn now dofntester is deprecated).
>>
>> Le 27 févr. 2018 21:45, "Chamikara Jayalath" <chamik...@google.com> a
>> écrit :
>>
>>> Looks like previous URL was broken. Created https://s.apache.org/
>>> beam-2.4.0-burndown.
>>>
>>> - Cham
>>>
>>> On Tue, Feb 27, 2018 at 12:22 PM Robert Bradshaw <rober...@google.com>
>>> wrote:
>>>
>>>> I'm planning on cutting the 2.4.0 release branch soon (tomorrow?). I
>>>> see 13
>>>> open issues on JIRA [1], none of which are labeled as blockers. If there
>>>> are any that cannot be bumped to the next release, let me know soon.
>>>>
>>>> - Robert
>>>>
>>>>
>>>> [1]
>>>> https://issues.apache.org/jira/browse/BEAM-3749?jql=
>>>> project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%
>>>> 22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
>>>>
>>>

Re: Beam 2.4.0

[Reuven Lax]

Ok, I see. This bug is specifically about Teardown behavior.


On Tue, Feb 27, 2018 at 1:33 PM Reuven Lax <re...@google.com> wrote:

> Can you explain "unusable?" We have hundreds of direct-runner tests that
> appear to still be running just fine. Is this a new regression, or old
> behavior?
>
>
> On Tue, Feb 27, 2018 at 1:30 PM Romain Manni-Bucau <rmannibu...@gmail.com>
> wrote:
>
>> Updated 3409 as blocker since it makes direct runner - and therefore any
>> test - unusable at all (leaks+unexpected wait times + retry strategy
>> required+no real alternative even for dofn now dofntester is deprecated).
>>
>> Le 27 févr. 2018 21:45, "Chamikara Jayalath" <chamik...@google.com> a
>> écrit :
>>
>>> Looks like previous URL was broken. Created
>>> https://s.apache.org/beam-2.4.0-burndown.
>>>
>>> - Cham
>>>
>>> On Tue, Feb 27, 2018 at 12:22 PM Robert Bradshaw <rober...@google.com>
>>> wrote:
>>>
>>>> I'm planning on cutting the 2.4.0 release branch soon (tomorrow?). I
>>>> see 13
>>>> open issues on JIRA [1], none of which are labeled as blockers. If there
>>>> are any that cannot be bumped to the next release, let me know soon.
>>>>
>>>> - Robert
>>>>
>>>>
>>>> [1]
>>>>
>>>> https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
>>>>
>>>

Re: Beam 2.4.0

[Reuven Lax]

Can you explain "unusable?" We have hundreds of direct-runner tests that
appear to still be running just fine. Is this a new regression, or old
behavior?


On Tue, Feb 27, 2018 at 1:30 PM Romain Manni-Bucau <rmannibu...@gmail.com>
wrote:

> Updated 3409 as blocker since it makes direct runner - and therefore any
> test - unusable at all (leaks+unexpected wait times + retry strategy
> required+no real alternative even for dofn now dofntester is deprecated).
>
> Le 27 févr. 2018 21:45, "Chamikara Jayalath" <chamik...@google.com> a
> écrit :
>
>> Looks like previous URL was broken. Created
>> https://s.apache.org/beam-2.4.0-burndown.
>>
>> - Cham
>>
>> On Tue, Feb 27, 2018 at 12:22 PM Robert Bradshaw <rober...@google.com>
>> wrote:
>>
>>> I'm planning on cutting the 2.4.0 release branch soon (tomorrow?). I see
>>> 13
>>> open issues on JIRA [1], none of which are labeled as blockers. If there
>>> are any that cannot be bumped to the next release, let me know soon.
>>>
>>> - Robert
>>>
>>>
>>> [1]
>>>
>>> https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
>>>
>>

Re: Beam 2.4.0

[Romain Manni-Bucau]

Updated 3409 as blocker since it makes direct runner - and therefore any
test - unusable at all (leaks+unexpected wait times + retry strategy
required+no real alternative even for dofn now dofntester is deprecated).

Le 27 févr. 2018 21:45, "Chamikara Jayalath" <chamik...@google.com> a
écrit :

> Looks like previous URL was broken. Created https://s.apache.org/
> beam-2.4.0-burndown.
>
> - Cham
>
> On Tue, Feb 27, 2018 at 12:22 PM Robert Bradshaw <rober...@google.com>
> wrote:
>
>> I'm planning on cutting the 2.4.0 release branch soon (tomorrow?). I see
>> 13
>> open issues on JIRA [1], none of which are labeled as blockers. If there
>> are any that cannot be bumped to the next release, let me know soon.
>>
>> - Robert
>>
>>
>> [1]
>> https://issues.apache.org/jira/browse/BEAM-3749?jql=
>> project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%
>> 22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
>>
>

Re: Beam 2.4.0

[Chamikara Jayalath]

Looks like previous URL was broken. Created
https://s.apache.org/beam-2.4.0-burndown.

- Cham

On Tue, Feb 27, 2018 at 12:22 PM Robert Bradshaw <rober...@google.com>
wrote:

> I'm planning on cutting the 2.4.0 release branch soon (tomorrow?). I see 13
> open issues on JIRA [1], none of which are labeled as blockers. If there
> are any that cannot be bumped to the next release, let me know soon.
>
> - Robert
>
>
> [1]
>
> https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
>

Beam 2.4.0

[Robert Bradshaw]

I'm planning on cutting the 2.4.0 release branch soon (tomorrow?). I see 13
open issues on JIRA [1], none of which are labeled as blockers. If there
are any that cannot be bumped to the next release, let me know soon.

- Robert


[1]
https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0

Re: Beam 2.4.0

[Romain Manni-Bucau]

done


Romain Manni-Bucau
@rmannibucau  |  Blog
 | Old Blog
 | Github  |
LinkedIn  | Book


2018-02-21 18:31 GMT+01:00 Kenneth Knowles :

> *are _not_ on the burndown :-)
>
> On Wed, Feb 21, 2018 at 9:31 AM, Kenneth Knowles  wrote:
>
>> Romain - it looks like these JIRA tickets are on on the 2.4.0 burndown.
>> Can you set their Fix Version field to make sure they are tracked and
>> triaged?
>>
>>
>> On Tue, Feb 20, 2018 at 10:22 PM, Reuven Lax  wrote:
>>
>>> I think it's fair to request that the reviewers of these PRs help with
>>> your effort to get them merged before the 2.4.0 cut. Existing comments on
>>> the PR imply that reviewers think the approaches are reasonable. Assuming
>>> that there's not too much work left to be done to address comments, there's
>>> a good chance of getting them in.
>>>
>>> Reuven
>>>
>>> On Tue, Feb 20, 2018 at 10:10 PM, Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 Ok

 In terms of what I'd like included, here is the list:

 1. https://github.com/apache/beam/pull/4412 (important to prevent
 regressions)
 2. https://github.com/apache/beam/pull/4674 (can need some more work
 but can break some api if we do, so current state is a functional trade
 off). On a more personal side Im blocked by this one for some features.
 3. https://github.com/apache/beam/pull/4372 (important cause doesnt
 make the execution deterministic depending your surefire config, IDE, main
 usage)



 Le 21 févr. 2018 01:29, "Reuven Lax"  a écrit :

> +1, this is keeping with an every-six weeks cadence.
>
> Romain, you can always target Jiras to this release, and then the
> release manager can decide on a case-by-case basis whether to make sure 
> the
> fix is included.
>
> On Tue, Feb 20, 2018 at 2:30 PM, Robert Bradshaw 
> wrote:
>
>> Yep. I am starting the "Let's do a 2.4.0 release" thread almost
>> exactly 6 weeks after JB first started the 2.3.0 release thread.
>>
>> On Tue, Feb 20, 2018 at 2:20 PM, Charles Chen  wrote:
>> > I would like to +1 the faster release cycle process JB and Robert
>> have been
>> > advocating and implementing, and thank JB for releasing 2.3.0
>> smoothly.
>> > When we block for specific features and increase the time between
>> releases,
>> > we increase the urgency for PR authors to push for their change to
>> go into
>> > an upcoming release, which is a feedback loop that results in our
>> releases
>> > taking months instead of weeks.  We should however try to get
>> pending PRs
>> > wrapped up.
>> >
>> > On Tue, Feb 20, 2018 at 2:15 PM Romain Manni-Bucau <
>> rmannibu...@gmail.com>
>> > wrote:
>> >>
>> >> Kind of agree but rythm was supposed to be 6 weeks IIRC, 2.3 is
>> just out
>> >> so 1 week is a bit fast IMHO.
>> >>
>> >> Le 20 févr. 2018 23:13, "Robert Bradshaw"  a
>> écrit :
>> >>>
>> >>> One of the main shifts that I think helped this release was
>> explicitly
>> >>> not being feature driven, rather releasing what's already in the
>> >>> branch. That doesn't mean it's not a good call to action to try
>> and
>> >>> get long-pending PRs or similar wrapped up.
>> >>>
>> >>> On Tue, Feb 20, 2018 at 2:10 PM, Romain Manni-Bucau
>> >>>  wrote:
>> >>> > There are a lot of long pending PR, would be good to merge them
>> before
>> >>> > 2.4.
>> >>> > Some are bringing tests for the 2.3 release which can be
>> critical to
>> >>> > include.
>> >>> >
>> >>> > Maybe we should list the pr and jira we want it before picking
>> a date?
>> >>> >
>> >>> > Le 20 févr. 2018 22:02, "Konstantinos Katsiapis" <
>> katsia...@google.com>
>> >>> > a
>> >>> > écrit :
>> >>> >>
>> >>> >> +1 since tf.transform 0.6 depends on Beam 2.4 and Tensorflow
>> 1.6 (and
>> >>> >> the
>> >>> >> latter already has an RC out, so we will likely be blocked on
>> Beam).
>> >>> >>
>> >>> >> On Tue, Feb 20, 2018 at 12:50 PM, Robert Bradshaw
>> >>> >> 
>> >>> >> wrote:
>> >>> >>>
>> >>> >>> Now that Beam 2.3.0 went out (and in record time, kudos to
>> all that
>> >>> >>> made this happen!) It'd be great to keep the ball rolling for
>> a
>> >>> >>> similarly well-executed 2.4. A lot has gone in [1] since we
>> made the
>> >>> >>> 2.3 cut, and to keep our cadence up I would propose a
>> time-based 

Re: Beam 2.4.0

[Kenneth Knowles]

Romain - it looks like these JIRA tickets are on on the 2.4.0 burndown. Can
you set their Fix Version field to make sure they are tracked and triaged?


On Tue, Feb 20, 2018 at 10:22 PM, Reuven Lax  wrote:

> I think it's fair to request that the reviewers of these PRs help with
> your effort to get them merged before the 2.4.0 cut. Existing comments on
> the PR imply that reviewers think the approaches are reasonable. Assuming
> that there's not too much work left to be done to address comments, there's
> a good chance of getting them in.
>
> Reuven
>
> On Tue, Feb 20, 2018 at 10:10 PM, Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>> Ok
>>
>> In terms of what I'd like included, here is the list:
>>
>> 1. https://github.com/apache/beam/pull/4412 (important to prevent
>> regressions)
>> 2. https://github.com/apache/beam/pull/4674 (can need some more work but
>> can break some api if we do, so current state is a functional trade off).
>> On a more personal side Im blocked by this one for some features.
>> 3. https://github.com/apache/beam/pull/4372 (important cause doesnt make
>> the execution deterministic depending your surefire config, IDE, main usage)
>>
>>
>>
>> Le 21 févr. 2018 01:29, "Reuven Lax"  a écrit :
>>
>>> +1, this is keeping with an every-six weeks cadence.
>>>
>>> Romain, you can always target Jiras to this release, and then the
>>> release manager can decide on a case-by-case basis whether to make sure the
>>> fix is included.
>>>
>>> On Tue, Feb 20, 2018 at 2:30 PM, Robert Bradshaw 
>>> wrote:
>>>
 Yep. I am starting the "Let's do a 2.4.0 release" thread almost
 exactly 6 weeks after JB first started the 2.3.0 release thread.

 On Tue, Feb 20, 2018 at 2:20 PM, Charles Chen  wrote:
 > I would like to +1 the faster release cycle process JB and Robert
 have been
 > advocating and implementing, and thank JB for releasing 2.3.0
 smoothly.
 > When we block for specific features and increase the time between
 releases,
 > we increase the urgency for PR authors to push for their change to go
 into
 > an upcoming release, which is a feedback loop that results in our
 releases
 > taking months instead of weeks.  We should however try to get pending
 PRs
 > wrapped up.
 >
 > On Tue, Feb 20, 2018 at 2:15 PM Romain Manni-Bucau <
 rmannibu...@gmail.com>
 > wrote:
 >>
 >> Kind of agree but rythm was supposed to be 6 weeks IIRC, 2.3 is just
 out
 >> so 1 week is a bit fast IMHO.
 >>
 >> Le 20 févr. 2018 23:13, "Robert Bradshaw"  a
 écrit :
 >>>
 >>> One of the main shifts that I think helped this release was
 explicitly
 >>> not being feature driven, rather releasing what's already in the
 >>> branch. That doesn't mean it's not a good call to action to try and
 >>> get long-pending PRs or similar wrapped up.
 >>>
 >>> On Tue, Feb 20, 2018 at 2:10 PM, Romain Manni-Bucau
 >>>  wrote:
 >>> > There are a lot of long pending PR, would be good to merge them
 before
 >>> > 2.4.
 >>> > Some are bringing tests for the 2.3 release which can be critical
 to
 >>> > include.
 >>> >
 >>> > Maybe we should list the pr and jira we want it before picking a
 date?
 >>> >
 >>> > Le 20 févr. 2018 22:02, "Konstantinos Katsiapis" <
 katsia...@google.com>
 >>> > a
 >>> > écrit :
 >>> >>
 >>> >> +1 since tf.transform 0.6 depends on Beam 2.4 and Tensorflow 1.6
 (and
 >>> >> the
 >>> >> latter already has an RC out, so we will likely be blocked on
 Beam).
 >>> >>
 >>> >> On Tue, Feb 20, 2018 at 12:50 PM, Robert Bradshaw
 >>> >> 
 >>> >> wrote:
 >>> >>>
 >>> >>> Now that Beam 2.3.0 went out (and in record time, kudos to all
 that
 >>> >>> made this happen!) It'd be great to keep the ball rolling for a
 >>> >>> similarly well-executed 2.4. A lot has gone in [1] since we
 made the
 >>> >>> 2.3 cut, and to keep our cadence up I would propose a
 time-based cut
 >>> >>> date early next week (say the 28th).
 >>> >>>
 >>> >>> I'll volunteer to do this release.
 >>> >>>
 >>> >>> [1] https://github.com/apache/beam
 /compare/release-2.3.0...master
 >>> >>
 >>> >>
 >>> >>
 >>> >>
 >>> >> --
 >>> >> Gus Katsiapis | Software Engineer | katsia...@google.com |
 >>> >> 650-918-7487

>>>
>>>
>

Re: Beam 2.4.0

[Kenneth Knowles]

*are _not_ on the burndown :-)

On Wed, Feb 21, 2018 at 9:31 AM, Kenneth Knowles  wrote:

> Romain - it looks like these JIRA tickets are on on the 2.4.0 burndown.
> Can you set their Fix Version field to make sure they are tracked and
> triaged?
>
>
> On Tue, Feb 20, 2018 at 10:22 PM, Reuven Lax  wrote:
>
>> I think it's fair to request that the reviewers of these PRs help with
>> your effort to get them merged before the 2.4.0 cut. Existing comments on
>> the PR imply that reviewers think the approaches are reasonable. Assuming
>> that there's not too much work left to be done to address comments, there's
>> a good chance of getting them in.
>>
>> Reuven
>>
>> On Tue, Feb 20, 2018 at 10:10 PM, Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> Ok
>>>
>>> In terms of what I'd like included, here is the list:
>>>
>>> 1. https://github.com/apache/beam/pull/4412 (important to prevent
>>> regressions)
>>> 2. https://github.com/apache/beam/pull/4674 (can need some more work
>>> but can break some api if we do, so current state is a functional trade
>>> off). On a more personal side Im blocked by this one for some features.
>>> 3. https://github.com/apache/beam/pull/4372 (important cause doesnt
>>> make the execution deterministic depending your surefire config, IDE, main
>>> usage)
>>>
>>>
>>>
>>> Le 21 févr. 2018 01:29, "Reuven Lax"  a écrit :
>>>
 +1, this is keeping with an every-six weeks cadence.

 Romain, you can always target Jiras to this release, and then the
 release manager can decide on a case-by-case basis whether to make sure the
 fix is included.

 On Tue, Feb 20, 2018 at 2:30 PM, Robert Bradshaw 
 wrote:

> Yep. I am starting the "Let's do a 2.4.0 release" thread almost
> exactly 6 weeks after JB first started the 2.3.0 release thread.
>
> On Tue, Feb 20, 2018 at 2:20 PM, Charles Chen  wrote:
> > I would like to +1 the faster release cycle process JB and Robert
> have been
> > advocating and implementing, and thank JB for releasing 2.3.0
> smoothly.
> > When we block for specific features and increase the time between
> releases,
> > we increase the urgency for PR authors to push for their change to
> go into
> > an upcoming release, which is a feedback loop that results in our
> releases
> > taking months instead of weeks.  We should however try to get
> pending PRs
> > wrapped up.
> >
> > On Tue, Feb 20, 2018 at 2:15 PM Romain Manni-Bucau <
> rmannibu...@gmail.com>
> > wrote:
> >>
> >> Kind of agree but rythm was supposed to be 6 weeks IIRC, 2.3 is
> just out
> >> so 1 week is a bit fast IMHO.
> >>
> >> Le 20 févr. 2018 23:13, "Robert Bradshaw"  a
> écrit :
> >>>
> >>> One of the main shifts that I think helped this release was
> explicitly
> >>> not being feature driven, rather releasing what's already in the
> >>> branch. That doesn't mean it's not a good call to action to try and
> >>> get long-pending PRs or similar wrapped up.
> >>>
> >>> On Tue, Feb 20, 2018 at 2:10 PM, Romain Manni-Bucau
> >>>  wrote:
> >>> > There are a lot of long pending PR, would be good to merge them
> before
> >>> > 2.4.
> >>> > Some are bringing tests for the 2.3 release which can be
> critical to
> >>> > include.
> >>> >
> >>> > Maybe we should list the pr and jira we want it before picking a
> date?
> >>> >
> >>> > Le 20 févr. 2018 22:02, "Konstantinos Katsiapis" <
> katsia...@google.com>
> >>> > a
> >>> > écrit :
> >>> >>
> >>> >> +1 since tf.transform 0.6 depends on Beam 2.4 and Tensorflow
> 1.6 (and
> >>> >> the
> >>> >> latter already has an RC out, so we will likely be blocked on
> Beam).
> >>> >>
> >>> >> On Tue, Feb 20, 2018 at 12:50 PM, Robert Bradshaw
> >>> >> 
> >>> >> wrote:
> >>> >>>
> >>> >>> Now that Beam 2.3.0 went out (and in record time, kudos to all
> that
> >>> >>> made this happen!) It'd be great to keep the ball rolling for a
> >>> >>> similarly well-executed 2.4. A lot has gone in [1] since we
> made the
> >>> >>> 2.3 cut, and to keep our cadence up I would propose a
> time-based cut
> >>> >>> date early next week (say the 28th).
> >>> >>>
> >>> >>> I'll volunteer to do this release.
> >>> >>>
> >>> >>> [1] https://github.com/apache/beam
> /compare/release-2.3.0...master
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >> --
> >>> >> Gus Katsiapis | Software Engineer | katsia...@google.com |
> >>> >> 650-918-7487
>


>>
>

Re: Beam 2.4.0

[Reuven Lax]

I think it's fair to request that the reviewers of these PRs help with your
effort to get them merged before the 2.4.0 cut. Existing comments on the PR
imply that reviewers think the approaches are reasonable. Assuming that
there's not too much work left to be done to address comments, there's a
good chance of getting them in.

Reuven

On Tue, Feb 20, 2018 at 10:10 PM, Romain Manni-Bucau 
wrote:

> Ok
>
> In terms of what I'd like included, here is the list:
>
> 1. https://github.com/apache/beam/pull/4412 (important to prevent
> regressions)
> 2. https://github.com/apache/beam/pull/4674 (can need some more work but
> can break some api if we do, so current state is a functional trade off).
> On a more personal side Im blocked by this one for some features.
> 3. https://github.com/apache/beam/pull/4372 (important cause doesnt make
> the execution deterministic depending your surefire config, IDE, main usage)
>
>
>
> Le 21 févr. 2018 01:29, "Reuven Lax"  a écrit :
>
>> +1, this is keeping with an every-six weeks cadence.
>>
>> Romain, you can always target Jiras to this release, and then the release
>> manager can decide on a case-by-case basis whether to make sure the fix is
>> included.
>>
>> On Tue, Feb 20, 2018 at 2:30 PM, Robert Bradshaw 
>> wrote:
>>
>>> Yep. I am starting the "Let's do a 2.4.0 release" thread almost
>>> exactly 6 weeks after JB first started the 2.3.0 release thread.
>>>
>>> On Tue, Feb 20, 2018 at 2:20 PM, Charles Chen  wrote:
>>> > I would like to +1 the faster release cycle process JB and Robert have
>>> been
>>> > advocating and implementing, and thank JB for releasing 2.3.0 smoothly.
>>> > When we block for specific features and increase the time between
>>> releases,
>>> > we increase the urgency for PR authors to push for their change to go
>>> into
>>> > an upcoming release, which is a feedback loop that results in our
>>> releases
>>> > taking months instead of weeks.  We should however try to get pending
>>> PRs
>>> > wrapped up.
>>> >
>>> > On Tue, Feb 20, 2018 at 2:15 PM Romain Manni-Bucau <
>>> rmannibu...@gmail.com>
>>> > wrote:
>>> >>
>>> >> Kind of agree but rythm was supposed to be 6 weeks IIRC, 2.3 is just
>>> out
>>> >> so 1 week is a bit fast IMHO.
>>> >>
>>> >> Le 20 févr. 2018 23:13, "Robert Bradshaw"  a
>>> écrit :
>>> >>>
>>> >>> One of the main shifts that I think helped this release was
>>> explicitly
>>> >>> not being feature driven, rather releasing what's already in the
>>> >>> branch. That doesn't mean it's not a good call to action to try and
>>> >>> get long-pending PRs or similar wrapped up.
>>> >>>
>>> >>> On Tue, Feb 20, 2018 at 2:10 PM, Romain Manni-Bucau
>>> >>>  wrote:
>>> >>> > There are a lot of long pending PR, would be good to merge them
>>> before
>>> >>> > 2.4.
>>> >>> > Some are bringing tests for the 2.3 release which can be critical
>>> to
>>> >>> > include.
>>> >>> >
>>> >>> > Maybe we should list the pr and jira we want it before picking a
>>> date?
>>> >>> >
>>> >>> > Le 20 févr. 2018 22:02, "Konstantinos Katsiapis" <
>>> katsia...@google.com>
>>> >>> > a
>>> >>> > écrit :
>>> >>> >>
>>> >>> >> +1 since tf.transform 0.6 depends on Beam 2.4 and Tensorflow 1.6
>>> (and
>>> >>> >> the
>>> >>> >> latter already has an RC out, so we will likely be blocked on
>>> Beam).
>>> >>> >>
>>> >>> >> On Tue, Feb 20, 2018 at 12:50 PM, Robert Bradshaw
>>> >>> >> 
>>> >>> >> wrote:
>>> >>> >>>
>>> >>> >>> Now that Beam 2.3.0 went out (and in record time, kudos to all
>>> that
>>> >>> >>> made this happen!) It'd be great to keep the ball rolling for a
>>> >>> >>> similarly well-executed 2.4. A lot has gone in [1] since we made
>>> the
>>> >>> >>> 2.3 cut, and to keep our cadence up I would propose a time-based
>>> cut
>>> >>> >>> date early next week (say the 28th).
>>> >>> >>>
>>> >>> >>> I'll volunteer to do this release.
>>> >>> >>>
>>> >>> >>> [1] https://github.com/apache/beam/compare/release-2.3.0...maste
>>> r
>>> >>> >>
>>> >>> >>
>>> >>> >>
>>> >>> >>
>>> >>> >> --
>>> >>> >> Gus Katsiapis | Software Engineer | katsia...@google.com |
>>> >>> >> 650-918-7487
>>>
>>
>>

Re: Beam 2.4.0

[Romain Manni-Bucau]

Ok

In terms of what I'd like included, here is the list:

1. https://github.com/apache/beam/pull/4412 (important to prevent
regressions)
2. https://github.com/apache/beam/pull/4674 (can need some more work but
can break some api if we do, so current state is a functional trade off).
On a more personal side Im blocked by this one for some features.
3. https://github.com/apache/beam/pull/4372 (important cause doesnt make
the execution deterministic depending your surefire config, IDE, main usage)



Le 21 févr. 2018 01:29, "Reuven Lax"  a écrit :

> +1, this is keeping with an every-six weeks cadence.
>
> Romain, you can always target Jiras to this release, and then the release
> manager can decide on a case-by-case basis whether to make sure the fix is
> included.
>
> On Tue, Feb 20, 2018 at 2:30 PM, Robert Bradshaw 
> wrote:
>
>> Yep. I am starting the "Let's do a 2.4.0 release" thread almost
>> exactly 6 weeks after JB first started the 2.3.0 release thread.
>>
>> On Tue, Feb 20, 2018 at 2:20 PM, Charles Chen  wrote:
>> > I would like to +1 the faster release cycle process JB and Robert have
>> been
>> > advocating and implementing, and thank JB for releasing 2.3.0 smoothly.
>> > When we block for specific features and increase the time between
>> releases,
>> > we increase the urgency for PR authors to push for their change to go
>> into
>> > an upcoming release, which is a feedback loop that results in our
>> releases
>> > taking months instead of weeks.  We should however try to get pending
>> PRs
>> > wrapped up.
>> >
>> > On Tue, Feb 20, 2018 at 2:15 PM Romain Manni-Bucau <
>> rmannibu...@gmail.com>
>> > wrote:
>> >>
>> >> Kind of agree but rythm was supposed to be 6 weeks IIRC, 2.3 is just
>> out
>> >> so 1 week is a bit fast IMHO.
>> >>
>> >> Le 20 févr. 2018 23:13, "Robert Bradshaw"  a
>> écrit :
>> >>>
>> >>> One of the main shifts that I think helped this release was explicitly
>> >>> not being feature driven, rather releasing what's already in the
>> >>> branch. That doesn't mean it's not a good call to action to try and
>> >>> get long-pending PRs or similar wrapped up.
>> >>>
>> >>> On Tue, Feb 20, 2018 at 2:10 PM, Romain Manni-Bucau
>> >>>  wrote:
>> >>> > There are a lot of long pending PR, would be good to merge them
>> before
>> >>> > 2.4.
>> >>> > Some are bringing tests for the 2.3 release which can be critical to
>> >>> > include.
>> >>> >
>> >>> > Maybe we should list the pr and jira we want it before picking a
>> date?
>> >>> >
>> >>> > Le 20 févr. 2018 22:02, "Konstantinos Katsiapis" <
>> katsia...@google.com>
>> >>> > a
>> >>> > écrit :
>> >>> >>
>> >>> >> +1 since tf.transform 0.6 depends on Beam 2.4 and Tensorflow 1.6
>> (and
>> >>> >> the
>> >>> >> latter already has an RC out, so we will likely be blocked on
>> Beam).
>> >>> >>
>> >>> >> On Tue, Feb 20, 2018 at 12:50 PM, Robert Bradshaw
>> >>> >> 
>> >>> >> wrote:
>> >>> >>>
>> >>> >>> Now that Beam 2.3.0 went out (and in record time, kudos to all
>> that
>> >>> >>> made this happen!) It'd be great to keep the ball rolling for a
>> >>> >>> similarly well-executed 2.4. A lot has gone in [1] since we made
>> the
>> >>> >>> 2.3 cut, and to keep our cadence up I would propose a time-based
>> cut
>> >>> >>> date early next week (say the 28th).
>> >>> >>>
>> >>> >>> I'll volunteer to do this release.
>> >>> >>>
>> >>> >>> [1] https://github.com/apache/beam/compare/release-2.3.0...master
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> --
>> >>> >> Gus Katsiapis | Software Engineer | katsia...@google.com |
>> >>> >> 650-918-7487
>>
>
>

Re: Beam 2.4.0

[Reuven Lax]

+1, this is keeping with an every-six weeks cadence.

Romain, you can always target Jiras to this release, and then the release
manager can decide on a case-by-case basis whether to make sure the fix is
included.

On Tue, Feb 20, 2018 at 2:30 PM, Robert Bradshaw 
wrote:

> Yep. I am starting the "Let's do a 2.4.0 release" thread almost
> exactly 6 weeks after JB first started the 2.3.0 release thread.
>
> On Tue, Feb 20, 2018 at 2:20 PM, Charles Chen  wrote:
> > I would like to +1 the faster release cycle process JB and Robert have
> been
> > advocating and implementing, and thank JB for releasing 2.3.0 smoothly.
> > When we block for specific features and increase the time between
> releases,
> > we increase the urgency for PR authors to push for their change to go
> into
> > an upcoming release, which is a feedback loop that results in our
> releases
> > taking months instead of weeks.  We should however try to get pending PRs
> > wrapped up.
> >
> > On Tue, Feb 20, 2018 at 2:15 PM Romain Manni-Bucau <
> rmannibu...@gmail.com>
> > wrote:
> >>
> >> Kind of agree but rythm was supposed to be 6 weeks IIRC, 2.3 is just out
> >> so 1 week is a bit fast IMHO.
> >>
> >> Le 20 févr. 2018 23:13, "Robert Bradshaw"  a
> écrit :
> >>>
> >>> One of the main shifts that I think helped this release was explicitly
> >>> not being feature driven, rather releasing what's already in the
> >>> branch. That doesn't mean it's not a good call to action to try and
> >>> get long-pending PRs or similar wrapped up.
> >>>
> >>> On Tue, Feb 20, 2018 at 2:10 PM, Romain Manni-Bucau
> >>>  wrote:
> >>> > There are a lot of long pending PR, would be good to merge them
> before
> >>> > 2.4.
> >>> > Some are bringing tests for the 2.3 release which can be critical to
> >>> > include.
> >>> >
> >>> > Maybe we should list the pr and jira we want it before picking a
> date?
> >>> >
> >>> > Le 20 févr. 2018 22:02, "Konstantinos Katsiapis" <
> katsia...@google.com>
> >>> > a
> >>> > écrit :
> >>> >>
> >>> >> +1 since tf.transform 0.6 depends on Beam 2.4 and Tensorflow 1.6
> (and
> >>> >> the
> >>> >> latter already has an RC out, so we will likely be blocked on Beam).
> >>> >>
> >>> >> On Tue, Feb 20, 2018 at 12:50 PM, Robert Bradshaw
> >>> >> 
> >>> >> wrote:
> >>> >>>
> >>> >>> Now that Beam 2.3.0 went out (and in record time, kudos to all that
> >>> >>> made this happen!) It'd be great to keep the ball rolling for a
> >>> >>> similarly well-executed 2.4. A lot has gone in [1] since we made
> the
> >>> >>> 2.3 cut, and to keep our cadence up I would propose a time-based
> cut
> >>> >>> date early next week (say the 28th).
> >>> >>>
> >>> >>> I'll volunteer to do this release.
> >>> >>>
> >>> >>> [1] https://github.com/apache/beam/compare/release-2.3.0...master
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >> --
> >>> >> Gus Katsiapis | Software Engineer | katsia...@google.com |
> >>> >> 650-918-7487
>

Re: Beam 2.4.0

[Rafael Fernandez]

+1 on having release trains scheduled.

Romain: Do you have a list of PRs that could benefit from increased focus
if they want to make it on the upcoming train?


On Tue, Feb 20, 2018 at 3:30 PM Ahmet Altay  wrote:

> +1 for having regular release cycles. Finalizing a release takes time in
> the order of a few weeks and starting a new release soon after the previous
> one is a reliable way for having releases every 6 weeks.
>
> On Tue, Feb 20, 2018 at 2:30 PM, Robert Bradshaw 
> wrote:
>
>> Yep. I am starting the "Let's do a 2.4.0 release" thread almost
>> exactly 6 weeks after JB first started the 2.3.0 release thread.
>>
>> On Tue, Feb 20, 2018 at 2:20 PM, Charles Chen  wrote:
>> > I would like to +1 the faster release cycle process JB and Robert have
>> been
>> > advocating and implementing, and thank JB for releasing 2.3.0 smoothly.
>> > When we block for specific features and increase the time between
>> releases,
>> > we increase the urgency for PR authors to push for their change to go
>> into
>> > an upcoming release, which is a feedback loop that results in our
>> releases
>> > taking months instead of weeks.  We should however try to get pending
>> PRs
>> > wrapped up.
>> >
>> > On Tue, Feb 20, 2018 at 2:15 PM Romain Manni-Bucau <
>> rmannibu...@gmail.com>
>> > wrote:
>> >>
>> >> Kind of agree but rythm was supposed to be 6 weeks IIRC, 2.3 is just
>> out
>> >> so 1 week is a bit fast IMHO.
>> >>
>> >> Le 20 févr. 2018 23:13, "Robert Bradshaw"  a
>> écrit :
>> >>>
>> >>> One of the main shifts that I think helped this release was explicitly
>> >>> not being feature driven, rather releasing what's already in the
>> >>> branch. That doesn't mean it's not a good call to action to try and
>> >>> get long-pending PRs or similar wrapped up.
>> >>>
>> >>> On Tue, Feb 20, 2018 at 2:10 PM, Romain Manni-Bucau
>> >>>  wrote:
>> >>> > There are a lot of long pending PR, would be good to merge them
>> before
>> >>> > 2.4.
>> >>> > Some are bringing tests for the 2.3 release which can be critical to
>> >>> > include.
>> >>> >
>> >>> > Maybe we should list the pr and jira we want it before picking a
>> date?
>> >>> >
>> >>> > Le 20 févr. 2018 22:02, "Konstantinos Katsiapis" <
>> katsia...@google.com>
>> >>> > a
>> >>> > écrit :
>> >>> >>
>> >>> >> +1 since tf.transform 0.6 depends on Beam 2.4 and Tensorflow 1.6
>> (and
>> >>> >> the
>> >>> >> latter already has an RC out, so we will likely be blocked on
>> Beam).
>> >>> >>
>> >>> >> On Tue, Feb 20, 2018 at 12:50 PM, Robert Bradshaw
>> >>> >> 
>> >>> >> wrote:
>> >>> >>>
>> >>> >>> Now that Beam 2.3.0 went out (and in record time, kudos to all
>> that
>> >>> >>> made this happen!) It'd be great to keep the ball rolling for a
>> >>> >>> similarly well-executed 2.4. A lot has gone in [1] since we made
>> the
>> >>> >>> 2.3 cut, and to keep our cadence up I would propose a time-based
>> cut
>> >>> >>> date early next week (say the 28th).
>> >>> >>>
>> >>> >>> I'll volunteer to do this release.
>> >>> >>>
>> >>> >>> [1] https://github.com/apache/beam/compare/release-2.3.0...master
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> --
>> >>> >> Gus Katsiapis | Software Engineer | katsia...@google.com |
>> >>> >> 650-918-7487
>>
>
>


smime.p7s
Description: S/MIME Cryptographic Signature

Re: Beam 2.4.0

[Ahmet Altay]

+1 for having regular release cycles. Finalizing a release takes time in
the order of a few weeks and starting a new release soon after the previous
one is a reliable way for having releases every 6 weeks.

On Tue, Feb 20, 2018 at 2:30 PM, Robert Bradshaw 
wrote:

> Yep. I am starting the "Let's do a 2.4.0 release" thread almost
> exactly 6 weeks after JB first started the 2.3.0 release thread.
>
> On Tue, Feb 20, 2018 at 2:20 PM, Charles Chen  wrote:
> > I would like to +1 the faster release cycle process JB and Robert have
> been
> > advocating and implementing, and thank JB for releasing 2.3.0 smoothly.
> > When we block for specific features and increase the time between
> releases,
> > we increase the urgency for PR authors to push for their change to go
> into
> > an upcoming release, which is a feedback loop that results in our
> releases
> > taking months instead of weeks.  We should however try to get pending PRs
> > wrapped up.
> >
> > On Tue, Feb 20, 2018 at 2:15 PM Romain Manni-Bucau <
> rmannibu...@gmail.com>
> > wrote:
> >>
> >> Kind of agree but rythm was supposed to be 6 weeks IIRC, 2.3 is just out
> >> so 1 week is a bit fast IMHO.
> >>
> >> Le 20 févr. 2018 23:13, "Robert Bradshaw"  a
> écrit :
> >>>
> >>> One of the main shifts that I think helped this release was explicitly
> >>> not being feature driven, rather releasing what's already in the
> >>> branch. That doesn't mean it's not a good call to action to try and
> >>> get long-pending PRs or similar wrapped up.
> >>>
> >>> On Tue, Feb 20, 2018 at 2:10 PM, Romain Manni-Bucau
> >>>  wrote:
> >>> > There are a lot of long pending PR, would be good to merge them
> before
> >>> > 2.4.
> >>> > Some are bringing tests for the 2.3 release which can be critical to
> >>> > include.
> >>> >
> >>> > Maybe we should list the pr and jira we want it before picking a
> date?
> >>> >
> >>> > Le 20 févr. 2018 22:02, "Konstantinos Katsiapis" <
> katsia...@google.com>
> >>> > a
> >>> > écrit :
> >>> >>
> >>> >> +1 since tf.transform 0.6 depends on Beam 2.4 and Tensorflow 1.6
> (and
> >>> >> the
> >>> >> latter already has an RC out, so we will likely be blocked on Beam).
> >>> >>
> >>> >> On Tue, Feb 20, 2018 at 12:50 PM, Robert Bradshaw
> >>> >> 
> >>> >> wrote:
> >>> >>>
> >>> >>> Now that Beam 2.3.0 went out (and in record time, kudos to all that
> >>> >>> made this happen!) It'd be great to keep the ball rolling for a
> >>> >>> similarly well-executed 2.4. A lot has gone in [1] since we made
> the
> >>> >>> 2.3 cut, and to keep our cadence up I would propose a time-based
> cut
> >>> >>> date early next week (say the 28th).
> >>> >>>
> >>> >>> I'll volunteer to do this release.
> >>> >>>
> >>> >>> [1] https://github.com/apache/beam/compare/release-2.3.0...master
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >> --
> >>> >> Gus Katsiapis | Software Engineer | katsia...@google.com |
> >>> >> 650-918-7487
>

Re: Beam 2.4.0

[Robert Bradshaw]

Yep. I am starting the "Let's do a 2.4.0 release" thread almost
exactly 6 weeks after JB first started the 2.3.0 release thread.

On Tue, Feb 20, 2018 at 2:20 PM, Charles Chen  wrote:
> I would like to +1 the faster release cycle process JB and Robert have been
> advocating and implementing, and thank JB for releasing 2.3.0 smoothly.
> When we block for specific features and increase the time between releases,
> we increase the urgency for PR authors to push for their change to go into
> an upcoming release, which is a feedback loop that results in our releases
> taking months instead of weeks.  We should however try to get pending PRs
> wrapped up.
>
> On Tue, Feb 20, 2018 at 2:15 PM Romain Manni-Bucau 
> wrote:
>>
>> Kind of agree but rythm was supposed to be 6 weeks IIRC, 2.3 is just out
>> so 1 week is a bit fast IMHO.
>>
>> Le 20 févr. 2018 23:13, "Robert Bradshaw"  a écrit :
>>>
>>> One of the main shifts that I think helped this release was explicitly
>>> not being feature driven, rather releasing what's already in the
>>> branch. That doesn't mean it's not a good call to action to try and
>>> get long-pending PRs or similar wrapped up.
>>>
>>> On Tue, Feb 20, 2018 at 2:10 PM, Romain Manni-Bucau
>>>  wrote:
>>> > There are a lot of long pending PR, would be good to merge them before
>>> > 2.4.
>>> > Some are bringing tests for the 2.3 release which can be critical to
>>> > include.
>>> >
>>> > Maybe we should list the pr and jira we want it before picking a date?
>>> >
>>> > Le 20 févr. 2018 22:02, "Konstantinos Katsiapis" 
>>> > a
>>> > écrit :
>>> >>
>>> >> +1 since tf.transform 0.6 depends on Beam 2.4 and Tensorflow 1.6 (and
>>> >> the
>>> >> latter already has an RC out, so we will likely be blocked on Beam).
>>> >>
>>> >> On Tue, Feb 20, 2018 at 12:50 PM, Robert Bradshaw
>>> >> 
>>> >> wrote:
>>> >>>
>>> >>> Now that Beam 2.3.0 went out (and in record time, kudos to all that
>>> >>> made this happen!) It'd be great to keep the ball rolling for a
>>> >>> similarly well-executed 2.4. A lot has gone in [1] since we made the
>>> >>> 2.3 cut, and to keep our cadence up I would propose a time-based cut
>>> >>> date early next week (say the 28th).
>>> >>>
>>> >>> I'll volunteer to do this release.
>>> >>>
>>> >>> [1] https://github.com/apache/beam/compare/release-2.3.0...master
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Gus Katsiapis | Software Engineer | katsia...@google.com |
>>> >> 650-918-7487

Re: Beam 2.4.0

[Charles Chen]

I would like to +1 the faster release cycle process JB and Robert have been
advocating and implementing, and thank JB for releasing 2.3.0 smoothly.
When we block for specific features and increase the time between releases,
we increase the urgency for PR authors to push for their change to go into
an upcoming release, which is a feedback loop that results in our releases
taking months instead of weeks.  We should however try to get pending PRs
wrapped up.

On Tue, Feb 20, 2018 at 2:15 PM Romain Manni-Bucau 
wrote:

> Kind of agree but rythm was supposed to be 6 weeks IIRC, 2.3 is just out
> so 1 week is a bit fast IMHO.
>
> Le 20 févr. 2018 23:13, "Robert Bradshaw"  a écrit :
>
>> One of the main shifts that I think helped this release was explicitly
>> not being feature driven, rather releasing what's already in the
>> branch. That doesn't mean it's not a good call to action to try and
>> get long-pending PRs or similar wrapped up.
>>
>> On Tue, Feb 20, 2018 at 2:10 PM, Romain Manni-Bucau
>>  wrote:
>> > There are a lot of long pending PR, would be good to merge them before
>> 2.4.
>> > Some are bringing tests for the 2.3 release which can be critical to
>> > include.
>> >
>> > Maybe we should list the pr and jira we want it before picking a date?
>> >
>> > Le 20 févr. 2018 22:02, "Konstantinos Katsiapis" 
>> a
>> > écrit :
>> >>
>> >> +1 since tf.transform 0.6 depends on Beam 2.4 and Tensorflow 1.6 (and
>> the
>> >> latter already has an RC out, so we will likely be blocked on Beam).
>> >>
>> >> On Tue, Feb 20, 2018 at 12:50 PM, Robert Bradshaw > >
>> >> wrote:
>> >>>
>> >>> Now that Beam 2.3.0 went out (and in record time, kudos to all that
>> >>> made this happen!) It'd be great to keep the ball rolling for a
>> >>> similarly well-executed 2.4. A lot has gone in [1] since we made the
>> >>> 2.3 cut, and to keep our cadence up I would propose a time-based cut
>> >>> date early next week (say the 28th).
>> >>>
>> >>> I'll volunteer to do this release.
>> >>>
>> >>> [1] https://github.com/apache/beam/compare/release-2.3.0...master
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> Gus Katsiapis | Software Engineer | katsia...@google.com |
>> 650-918-7487 <(650)%20918-7487>
>>
>

Re: Beam 2.4.0

[Romain Manni-Bucau]

Kind of agree but rythm was supposed to be 6 weeks IIRC, 2.3 is just out so
1 week is a bit fast IMHO.

Le 20 févr. 2018 23:13, "Robert Bradshaw"  a écrit :

> One of the main shifts that I think helped this release was explicitly
> not being feature driven, rather releasing what's already in the
> branch. That doesn't mean it's not a good call to action to try and
> get long-pending PRs or similar wrapped up.
>
> On Tue, Feb 20, 2018 at 2:10 PM, Romain Manni-Bucau
>  wrote:
> > There are a lot of long pending PR, would be good to merge them before
> 2.4.
> > Some are bringing tests for the 2.3 release which can be critical to
> > include.
> >
> > Maybe we should list the pr and jira we want it before picking a date?
> >
> > Le 20 févr. 2018 22:02, "Konstantinos Katsiapis" 
> a
> > écrit :
> >>
> >> +1 since tf.transform 0.6 depends on Beam 2.4 and Tensorflow 1.6 (and
> the
> >> latter already has an RC out, so we will likely be blocked on Beam).
> >>
> >> On Tue, Feb 20, 2018 at 12:50 PM, Robert Bradshaw 
> >> wrote:
> >>>
> >>> Now that Beam 2.3.0 went out (and in record time, kudos to all that
> >>> made this happen!) It'd be great to keep the ball rolling for a
> >>> similarly well-executed 2.4. A lot has gone in [1] since we made the
> >>> 2.3 cut, and to keep our cadence up I would propose a time-based cut
> >>> date early next week (say the 28th).
> >>>
> >>> I'll volunteer to do this release.
> >>>
> >>> [1] https://github.com/apache/beam/compare/release-2.3.0...master
> >>
> >>
> >>
> >>
> >> --
> >> Gus Katsiapis | Software Engineer | katsia...@google.com | 650-918-7487
>

Re: Beam 2.4.0

[Robert Bradshaw]

One of the main shifts that I think helped this release was explicitly
not being feature driven, rather releasing what's already in the
branch. That doesn't mean it's not a good call to action to try and
get long-pending PRs or similar wrapped up.

On Tue, Feb 20, 2018 at 2:10 PM, Romain Manni-Bucau
 wrote:
> There are a lot of long pending PR, would be good to merge them before 2.4.
> Some are bringing tests for the 2.3 release which can be critical to
> include.
>
> Maybe we should list the pr and jira we want it before picking a date?
>
> Le 20 févr. 2018 22:02, "Konstantinos Katsiapis"  a
> écrit :
>>
>> +1 since tf.transform 0.6 depends on Beam 2.4 and Tensorflow 1.6 (and the
>> latter already has an RC out, so we will likely be blocked on Beam).
>>
>> On Tue, Feb 20, 2018 at 12:50 PM, Robert Bradshaw 
>> wrote:
>>>
>>> Now that Beam 2.3.0 went out (and in record time, kudos to all that
>>> made this happen!) It'd be great to keep the ball rolling for a
>>> similarly well-executed 2.4. A lot has gone in [1] since we made the
>>> 2.3 cut, and to keep our cadence up I would propose a time-based cut
>>> date early next week (say the 28th).
>>>
>>> I'll volunteer to do this release.
>>>
>>> [1] https://github.com/apache/beam/compare/release-2.3.0...master
>>
>>
>>
>>
>> --
>> Gus Katsiapis | Software Engineer | katsia...@google.com | 650-918-7487

Re: Beam 2.4.0

[Romain Manni-Bucau]

There are a lot of long pending PR, would be good to merge them before 2.4.
Some are bringing tests for the 2.3 release which can be critical to
include.

Maybe we should list the pr and jira we want it before picking a date?

Le 20 févr. 2018 22:02, "Konstantinos Katsiapis"  a
écrit :

> +1 since tf.transform  0.6
> depends on Beam 2.4 and Tensorflow 1.6 (and the latter already has an RC
> out, so we will likely be blocked on Beam).
>
> On Tue, Feb 20, 2018 at 12:50 PM, Robert Bradshaw 
> wrote:
>
>> Now that Beam 2.3.0 went out (and in record time, kudos to all that
>> made this happen!) It'd be great to keep the ball rolling for a
>> similarly well-executed 2.4. A lot has gone in [1] since we made the
>> 2.3 cut, and to keep our cadence up I would propose a time-based cut
>> date early next week (say the 28th).
>>
>> I'll volunteer to do this release.
>>
>> [1] https://github.com/apache/beam/compare/release-2.3.0...master
>>
>
>
>
> --
> Gus Katsiapis | Software Engineer | katsia...@google.com | 650-918-7487
>

Re: Beam 2.4.0

[Konstantinos Katsiapis]

+1 since tf.transform  0.6 depends
on Beam 2.4 and Tensorflow 1.6 (and the latter already has an RC out, so we
will likely be blocked on Beam).

On Tue, Feb 20, 2018 at 12:50 PM, Robert Bradshaw 
wrote:

> Now that Beam 2.3.0 went out (and in record time, kudos to all that
> made this happen!) It'd be great to keep the ball rolling for a
> similarly well-executed 2.4. A lot has gone in [1] since we made the
> 2.3 cut, and to keep our cadence up I would propose a time-based cut
> date early next week (say the 28th).
>
> I'll volunteer to do this release.
>
> [1] https://github.com/apache/beam/compare/release-2.3.0...master
>



-- 
Gus Katsiapis | Software Engineer | katsia...@google.com | 650-918-7487

Beam 2.4.0

[Robert Bradshaw]

Now that Beam 2.3.0 went out (and in record time, kudos to all that
made this happen!) It'd be great to keep the ball rolling for a
similarly well-executed 2.4. A lot has gone in [1] since we made the
2.3 cut, and to keep our cadence up I would propose a time-based cut
date early next week (say the 28th).

I'll volunteer to do this release.

[1] https://github.com/apache/beam/compare/release-2.3.0...master