subject:"\[VOTE\] Release 2.4.0, release candidate #2"

Re: [VOTE] Release 2.4.0, release candidate #2

2018-03-15 Thread Robert Bradshaw

Just to give an update on this, I plan on creating an RC3 as soon as I get
the dataflow containers rebuilt. (Been busy with the beam summit among
other things.)

Except for direct runner fixes and the dataflow tags, I expect it to be the
same as RC2,  so testing on flink/apex/spark done now would be applicable
to the next RC.

https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit?ts=5a1c7310#gid=1663314475
.


On Fri, Mar 9, 2018 at 4:24 PM Charles Chen  wrote:

> Thank you Valentyn for reporting this.  I have traced the issue back to
> https://github.com/apache/beam/pull/4666, so I have sent out a PR to fix:
> https://github.com/apache/beam/pull/4846.
>
> On Fri, Mar 9, 2018 at 2:17 PM, Valentyn Tymofieiev 
> wrote:
>
>> -1.
>>
>> Checked Python Quickstarts (Passed) and Python MobileGaming on
>> DirectRunner. I observe an issue in BQ sink for hourly teams score example:
>>
>> https://issues.apache.org/jira/browse/BEAM-3824
>>
>> On Fri, Mar 9, 2018 at 10:49 AM, Lukasz Cwik  wrote:
>>
>>> I checked that word count quickstarts (except Dataflow) worked for RC2
>>> to hopefully prevent an RC4.
>>>
>>> On Fri, Mar 9, 2018 at 10:29 AM, Robert Bradshaw 
>>> wrote:
>>>
 Thanks, Alan, for pointing this out. I see this now, and it looks like I
 need to finish building the dataflow workers so they have something to
 point to. I will do this and release an RC3 once that's ready.

 In the meantime, it'd be great if we could validate everything else
 about
 this RC such that when this on-line, dataflow-only change is out we
 won't
 have any further surprises. I see Luke went through the Java Quickstart
 examples, thanks!


 On Thu, Mar 8, 2018 at 3:48 PM Lukasz Cwik  wrote:

 > Yes, the release guide has a segment "Update release specific
 configurations" that has a tidbit about this.

 > On Thu, Mar 8, 2018 at 3:45 PM, Alan Myrvold 
 wrote:

 >> The dataflow java worker version wasn't updated on the branch as in
 past
 releases ... should it be?
 >> https://issues.apache.org/jira/browse/BEAM-3815


 >> On Thu, Mar 8, 2018 at 1:40 PM Romain Manni-Bucau <
 rmannibu...@gmail.com>
 wrote:

 >>> Can still be provided as a generic one (like the an offset or key
 based
 one) but good enough for now, right, was just surprising to not see it
 when
 checking the breakage.

 >>> Le 8 mars 2018 22:05, "Eugene Kirpichov"  a
 écrit
 :

 >>> All SDF-related method annotations in DoFn are marked
 @Experimental. I
 guess that should apply to RestrictionTracker too, but I wouldn't be too
 worried about that, since it only makes sense to use in the context of
 those methods.

 >>> On Thu, Mar 8, 2018 at 12:36 PM Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

  Hmm, does sdf api misses some @Experimental then?

  To clarify: for waitUntilFinish I'm fine with the 2.4 as this but
 cant
 +1 or +0 since none of my tests pass reliably in current state without a
 retry strategy making the call useless.

  Le 8 mars 2018 21:02, "Reuven Lax"  a écrit :

 > Does Nexmark use SerializableCoder?


 > On Thu, Mar 8, 2018 at 10:42 AM Robert Bradshaw <
 rober...@google.com>
 wrote:

 >> I put the validation checklist spreadsheet is up at

 https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit?ts=5a1c7310#gid=1663314475

 >> Regarding the direct runner regression on query 10, this is
 understandable given how mutation detection has been changed for
 serializable coders (and should be tracked, probably fixed by avoiding
 SerializableCoder). It should not affect other runners. Could you file a
 bug?

 >> Regarding waitUntilFinish, this is a bug but not a blocker--it's
 been this way since teardown was introduced. There are many
 nice-to-haves
 that one could merge from master to the release branch, but we've seen
 where that trend leads.

 >> Regarding the backwards incompatible changes in restriction
 tracker,
 this is (as I understand it) a change to the experimental SDF API.
 Eugene,
 do you want to comment on this?



 >> On Thu, Mar 8, 2018 at 2:07 AM Ismaël Mejía 
 wrote:

 >>> I confirm that the new release fixes both problems reported
 previously:

 >>> - python package name
 >>> - nexmark query 10 mutability issue with the direct runner.

 >>> One extra regression is that the the fix produced a way longer
 >>> execution time on the query.

Re: [VOTE] Release 2.4.0, release candidate #2

2018-03-09 Thread Charles Chen

Thank you Valentyn for reporting this.  I have traced the issue back to
https://github.com/apache/beam/pull/4666, so I have sent out a PR to fix:
https://github.com/apache/beam/pull/4846.

On Fri, Mar 9, 2018 at 2:17 PM, Valentyn Tymofieiev 
wrote:

> -1.
>
> Checked Python Quickstarts (Passed) and Python MobileGaming on
> DirectRunner. I observe an issue in BQ sink for hourly teams score example:
>
> https://issues.apache.org/jira/browse/BEAM-3824
>
> On Fri, Mar 9, 2018 at 10:49 AM, Lukasz Cwik  wrote:
>
>> I checked that word count quickstarts (except Dataflow) worked for RC2 to
>> hopefully prevent an RC4.
>>
>> On Fri, Mar 9, 2018 at 10:29 AM, Robert Bradshaw 
>> wrote:
>>
>>> Thanks, Alan, for pointing this out. I see this now, and it looks like I
>>> need to finish building the dataflow workers so they have something to
>>> point to. I will do this and release an RC3 once that's ready.
>>>
>>> In the meantime, it'd be great if we could validate everything else about
>>> this RC such that when this on-line, dataflow-only change is out we won't
>>> have any further surprises. I see Luke went through the Java Quickstart
>>> examples, thanks!
>>>
>>>
>>> On Thu, Mar 8, 2018 at 3:48 PM Lukasz Cwik  wrote:
>>>
>>> > Yes, the release guide has a segment "Update release specific
>>> configurations" that has a tidbit about this.
>>>
>>> > On Thu, Mar 8, 2018 at 3:45 PM, Alan Myrvold 
>>> wrote:
>>>
>>> >> The dataflow java worker version wasn't updated on the branch as in
>>> past
>>> releases ... should it be?
>>> >> https://issues.apache.org/jira/browse/BEAM-3815
>>>
>>>
>>> >> On Thu, Mar 8, 2018 at 1:40 PM Romain Manni-Bucau <
>>> rmannibu...@gmail.com>
>>> wrote:
>>>
>>> >>> Can still be provided as a generic one (like the an offset or key
>>> based
>>> one) but good enough for now, right, was just surprising to not see it
>>> when
>>> checking the breakage.
>>>
>>> >>> Le 8 mars 2018 22:05, "Eugene Kirpichov"  a
>>> écrit
>>> :
>>>
>>> >>> All SDF-related method annotations in DoFn are marked @Experimental.
>>> I
>>> guess that should apply to RestrictionTracker too, but I wouldn't be too
>>> worried about that, since it only makes sense to use in the context of
>>> those methods.
>>>
>>> >>> On Thu, Mar 8, 2018 at 12:36 PM Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
>>>  Hmm, does sdf api misses some @Experimental then?
>>>
>>>  To clarify: for waitUntilFinish I'm fine with the 2.4 as this but
>>> cant
>>> +1 or +0 since none of my tests pass reliably in current state without a
>>> retry strategy making the call useless.
>>>
>>>  Le 8 mars 2018 21:02, "Reuven Lax"  a écrit :
>>>
>>> > Does Nexmark use SerializableCoder?
>>>
>>>
>>> > On Thu, Mar 8, 2018 at 10:42 AM Robert Bradshaw <
>>> rober...@google.com>
>>> wrote:
>>>
>>> >> I put the validation checklist spreadsheet is up at
>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkS
>>> ZTR8AGqyNUM-oLFo_ZXBpJw/edit?ts=5a1c7310#gid=1663314475
>>>
>>> >> Regarding the direct runner regression on query 10, this is
>>> understandable given how mutation detection has been changed for
>>> serializable coders (and should be tracked, probably fixed by avoiding
>>> SerializableCoder). It should not affect other runners. Could you file a
>>> bug?
>>>
>>> >> Regarding waitUntilFinish, this is a bug but not a blocker--it's
>>> been this way since teardown was introduced. There are many nice-to-haves
>>> that one could merge from master to the release branch, but we've seen
>>> where that trend leads.
>>>
>>> >> Regarding the backwards incompatible changes in restriction
>>> tracker,
>>> this is (as I understand it) a change to the experimental SDF API.
>>> Eugene,
>>> do you want to comment on this?
>>>
>>>
>>>
>>> >> On Thu, Mar 8, 2018 at 2:07 AM Ismaël Mejía 
>>> wrote:
>>>
>>> >>> I confirm that the new release fixes both problems reported
>>> previously:
>>>
>>> >>> - python package name
>>> >>> - nexmark query 10 mutability issue with the direct runner.
>>>
>>> >>> One extra regression is that the the fix produced a way longer
>>> >>> execution time on the query.
>>> >>> Not sure if a blocker but worth tracking.
>>>
>>> >>> Query 10 - Batch/Bounded
>>> >>> Version  Runtime(sec)   Events(/sec)Results
>>> >>>2.3.0   3.627609.1  1
>>> >>>2.4.0  30.8 3244.3  1
>>>
>>> >>> Query 10 - Streaming/Unbounded
>>> >>> Version  Runtime(sec)   Events(/sec)Results
>>> >>>2.3.0   6.315873.0  1
>>> >>>2.4.0 101.1  989.4  1
>>>
>>> >>> On Thu, Mar 8, 2018 at 8:54 AM, Romain Manni-Bucau
>>> >>>  wrote:
>>> >>> > -1:
>>> >>> > a) still consider

Re: [VOTE] Release 2.4.0, release candidate #2

2018-03-09 Thread Valentyn Tymofieiev

-1.

Checked Python Quickstarts (Passed) and Python MobileGaming on
DirectRunner. I observe an issue in BQ sink for hourly teams score example:

https://issues.apache.org/jira/browse/BEAM-3824

On Fri, Mar 9, 2018 at 10:49 AM, Lukasz Cwik  wrote:

> I checked that word count quickstarts (except Dataflow) worked for RC2 to
> hopefully prevent an RC4.
>
> On Fri, Mar 9, 2018 at 10:29 AM, Robert Bradshaw 
> wrote:
>
>> Thanks, Alan, for pointing this out. I see this now, and it looks like I
>> need to finish building the dataflow workers so they have something to
>> point to. I will do this and release an RC3 once that's ready.
>>
>> In the meantime, it'd be great if we could validate everything else about
>> this RC such that when this on-line, dataflow-only change is out we won't
>> have any further surprises. I see Luke went through the Java Quickstart
>> examples, thanks!
>>
>>
>> On Thu, Mar 8, 2018 at 3:48 PM Lukasz Cwik  wrote:
>>
>> > Yes, the release guide has a segment "Update release specific
>> configurations" that has a tidbit about this.
>>
>> > On Thu, Mar 8, 2018 at 3:45 PM, Alan Myrvold 
>> wrote:
>>
>> >> The dataflow java worker version wasn't updated on the branch as in
>> past
>> releases ... should it be?
>> >> https://issues.apache.org/jira/browse/BEAM-3815
>>
>>
>> >> On Thu, Mar 8, 2018 at 1:40 PM Romain Manni-Bucau <
>> rmannibu...@gmail.com>
>> wrote:
>>
>> >>> Can still be provided as a generic one (like the an offset or key
>> based
>> one) but good enough for now, right, was just surprising to not see it
>> when
>> checking the breakage.
>>
>> >>> Le 8 mars 2018 22:05, "Eugene Kirpichov"  a
>> écrit
>> :
>>
>> >>> All SDF-related method annotations in DoFn are marked @Experimental. I
>> guess that should apply to RestrictionTracker too, but I wouldn't be too
>> worried about that, since it only makes sense to use in the context of
>> those methods.
>>
>> >>> On Thu, Mar 8, 2018 at 12:36 PM Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>  Hmm, does sdf api misses some @Experimental then?
>>
>>  To clarify: for waitUntilFinish I'm fine with the 2.4 as this but
>> cant
>> +1 or +0 since none of my tests pass reliably in current state without a
>> retry strategy making the call useless.
>>
>>  Le 8 mars 2018 21:02, "Reuven Lax"  a écrit :
>>
>> > Does Nexmark use SerializableCoder?
>>
>>
>> > On Thu, Mar 8, 2018 at 10:42 AM Robert Bradshaw <
>> rober...@google.com>
>> wrote:
>>
>> >> I put the validation checklist spreadsheet is up at
>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkS
>> ZTR8AGqyNUM-oLFo_ZXBpJw/edit?ts=5a1c7310#gid=1663314475
>>
>> >> Regarding the direct runner regression on query 10, this is
>> understandable given how mutation detection has been changed for
>> serializable coders (and should be tracked, probably fixed by avoiding
>> SerializableCoder). It should not affect other runners. Could you file a
>> bug?
>>
>> >> Regarding waitUntilFinish, this is a bug but not a blocker--it's
>> been this way since teardown was introduced. There are many nice-to-haves
>> that one could merge from master to the release branch, but we've seen
>> where that trend leads.
>>
>> >> Regarding the backwards incompatible changes in restriction
>> tracker,
>> this is (as I understand it) a change to the experimental SDF API. Eugene,
>> do you want to comment on this?
>>
>>
>>
>> >> On Thu, Mar 8, 2018 at 2:07 AM Ismaël Mejía 
>> wrote:
>>
>> >>> I confirm that the new release fixes both problems reported
>> previously:
>>
>> >>> - python package name
>> >>> - nexmark query 10 mutability issue with the direct runner.
>>
>> >>> One extra regression is that the the fix produced a way longer
>> >>> execution time on the query.
>> >>> Not sure if a blocker but worth tracking.
>>
>> >>> Query 10 - Batch/Bounded
>> >>> Version  Runtime(sec)   Events(/sec)Results
>> >>>2.3.0   3.627609.1  1
>> >>>2.4.0  30.8 3244.3  1
>>
>> >>> Query 10 - Streaming/Unbounded
>> >>> Version  Runtime(sec)   Events(/sec)Results
>> >>>2.3.0   6.315873.0  1
>> >>>2.4.0 101.1  989.4  1
>>
>> >>> On Thu, Mar 8, 2018 at 8:54 AM, Romain Manni-Bucau
>> >>>  wrote:
>> >>> > -1:
>> >>> > a) still consider waitUntilFinish broken and a big blocker
>> >>> > b) restrictiontracker api changed and is not backward compatible
>> >>> > (
>> https://github.com/apache/beam/commit/e0034314ad196d2274cef9
>> 831ed63e090bf4d4c1#diff-098d7247eb1e9d9423bfa2ae2da38a9d
>> )
>> >>> >
>> >>> > with workarounds and fixes for these two issues the other parts
>> work (spark,
>> >>> > flink, direct runner, java

Re: [VOTE] Release 2.4.0, release candidate #2

2018-03-09 Thread Lukasz Cwik

I checked that word count quickstarts (except Dataflow) worked for RC2 to
hopefully prevent an RC4.

On Fri, Mar 9, 2018 at 10:29 AM, Robert Bradshaw 
wrote:

> Thanks, Alan, for pointing this out. I see this now, and it looks like I
> need to finish building the dataflow workers so they have something to
> point to. I will do this and release an RC3 once that's ready.
>
> In the meantime, it'd be great if we could validate everything else about
> this RC such that when this on-line, dataflow-only change is out we won't
> have any further surprises. I see Luke went through the Java Quickstart
> examples, thanks!
>
>
> On Thu, Mar 8, 2018 at 3:48 PM Lukasz Cwik  wrote:
>
> > Yes, the release guide has a segment "Update release specific
> configurations" that has a tidbit about this.
>
> > On Thu, Mar 8, 2018 at 3:45 PM, Alan Myrvold 
> wrote:
>
> >> The dataflow java worker version wasn't updated on the branch as in past
> releases ... should it be?
> >> https://issues.apache.org/jira/browse/BEAM-3815
>
>
> >> On Thu, Mar 8, 2018 at 1:40 PM Romain Manni-Bucau <
> rmannibu...@gmail.com>
> wrote:
>
> >>> Can still be provided as a generic one (like the an offset or key based
> one) but good enough for now, right, was just surprising to not see it when
> checking the breakage.
>
> >>> Le 8 mars 2018 22:05, "Eugene Kirpichov"  a
> écrit
> :
>
> >>> All SDF-related method annotations in DoFn are marked @Experimental. I
> guess that should apply to RestrictionTracker too, but I wouldn't be too
> worried about that, since it only makes sense to use in the context of
> those methods.
>
> >>> On Thu, Mar 8, 2018 at 12:36 PM Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>  Hmm, does sdf api misses some @Experimental then?
>
>  To clarify: for waitUntilFinish I'm fine with the 2.4 as this but cant
> +1 or +0 since none of my tests pass reliably in current state without a
> retry strategy making the call useless.
>
>  Le 8 mars 2018 21:02, "Reuven Lax"  a écrit :
>
> > Does Nexmark use SerializableCoder?
>
>
> > On Thu, Mar 8, 2018 at 10:42 AM Robert Bradshaw  >
> wrote:
>
> >> I put the validation checklist spreadsheet is up at
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-
> oLFo_ZXBpJw/edit?ts=5a1c7310#gid=1663314475
>
> >> Regarding the direct runner regression on query 10, this is
> understandable given how mutation detection has been changed for
> serializable coders (and should be tracked, probably fixed by avoiding
> SerializableCoder). It should not affect other runners. Could you file a
> bug?
>
> >> Regarding waitUntilFinish, this is a bug but not a blocker--it's
> been this way since teardown was introduced. There are many nice-to-haves
> that one could merge from master to the release branch, but we've seen
> where that trend leads.
>
> >> Regarding the backwards incompatible changes in restriction tracker,
> this is (as I understand it) a change to the experimental SDF API. Eugene,
> do you want to comment on this?
>
>
>
> >> On Thu, Mar 8, 2018 at 2:07 AM Ismaël Mejía 
> wrote:
>
> >>> I confirm that the new release fixes both problems reported
> previously:
>
> >>> - python package name
> >>> - nexmark query 10 mutability issue with the direct runner.
>
> >>> One extra regression is that the the fix produced a way longer
> >>> execution time on the query.
> >>> Not sure if a blocker but worth tracking.
>
> >>> Query 10 - Batch/Bounded
> >>> Version  Runtime(sec)   Events(/sec)Results
> >>>2.3.0   3.627609.1  1
> >>>2.4.0  30.8 3244.3  1
>
> >>> Query 10 - Streaming/Unbounded
> >>> Version  Runtime(sec)   Events(/sec)Results
> >>>2.3.0   6.315873.0  1
> >>>2.4.0 101.1  989.4  1
>
> >>> On Thu, Mar 8, 2018 at 8:54 AM, Romain Manni-Bucau
> >>>  wrote:
> >>> > -1:
> >>> > a) still consider waitUntilFinish broken and a big blocker
> >>> > b) restrictiontracker api changed and is not backward compatible
> >>> > (
> https://github.com/apache/beam/commit/e0034314ad196d2274cef9831ed63e
> 090bf4d4c1#diff-098d7247eb1e9d9423bfa2ae2da38a9d
> )
> >>> >
> >>> > with workarounds and fixes for these two issues the other parts
> work (spark,
> >>> > flink, direct runner, java core) on my projects
> >>> >
> >>> >
> >>> >
> >>> > Romain Manni-Bucau
> >>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >>> >
> >>> > 2018-03-08 6:26 GMT+01:00 Robert Bradshaw :
> >>> >>
> >>> >> Hi everyone,
> >>> >>
> >>> >> Please review and vote on the release candidate #2 for the
> version 2.4.0,
> >>> >> as

Re: [VOTE] Release 2.4.0, release candidate #2

2018-03-09 Thread Robert Bradshaw

Thanks, Alan, for pointing this out. I see this now, and it looks like I
need to finish building the dataflow workers so they have something to
point to. I will do this and release an RC3 once that's ready.

In the meantime, it'd be great if we could validate everything else about
this RC such that when this on-line, dataflow-only change is out we won't
have any further surprises. I see Luke went through the Java Quickstart
examples, thanks!


On Thu, Mar 8, 2018 at 3:48 PM Lukasz Cwik  wrote:

> Yes, the release guide has a segment "Update release specific
configurations" that has a tidbit about this.

> On Thu, Mar 8, 2018 at 3:45 PM, Alan Myrvold  wrote:

>> The dataflow java worker version wasn't updated on the branch as in past
releases ... should it be?
>> https://issues.apache.org/jira/browse/BEAM-3815


>> On Thu, Mar 8, 2018 at 1:40 PM Romain Manni-Bucau 
wrote:

>>> Can still be provided as a generic one (like the an offset or key based
one) but good enough for now, right, was just surprising to not see it when
checking the breakage.

>>> Le 8 mars 2018 22:05, "Eugene Kirpichov"  a écrit
:

>>> All SDF-related method annotations in DoFn are marked @Experimental. I
guess that should apply to RestrictionTracker too, but I wouldn't be too
worried about that, since it only makes sense to use in the context of
those methods.

>>> On Thu, Mar 8, 2018 at 12:36 PM Romain Manni-Bucau <
rmannibu...@gmail.com> wrote:

 Hmm, does sdf api misses some @Experimental then?

 To clarify: for waitUntilFinish I'm fine with the 2.4 as this but cant
+1 or +0 since none of my tests pass reliably in current state without a
retry strategy making the call useless.

 Le 8 mars 2018 21:02, "Reuven Lax"  a écrit :

> Does Nexmark use SerializableCoder?


> On Thu, Mar 8, 2018 at 10:42 AM Robert Bradshaw 
wrote:

>> I put the validation checklist spreadsheet is up at
https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit?ts=5a1c7310#gid=1663314475

>> Regarding the direct runner regression on query 10, this is
understandable given how mutation detection has been changed for
serializable coders (and should be tracked, probably fixed by avoiding
SerializableCoder). It should not affect other runners. Could you file a
bug?

>> Regarding waitUntilFinish, this is a bug but not a blocker--it's
been this way since teardown was introduced. There are many nice-to-haves
that one could merge from master to the release branch, but we've seen
where that trend leads.

>> Regarding the backwards incompatible changes in restriction tracker,
this is (as I understand it) a change to the experimental SDF API. Eugene,
do you want to comment on this?



>> On Thu, Mar 8, 2018 at 2:07 AM Ismaël Mejía 
wrote:

>>> I confirm that the new release fixes both problems reported
previously:

>>> - python package name
>>> - nexmark query 10 mutability issue with the direct runner.

>>> One extra regression is that the the fix produced a way longer
>>> execution time on the query.
>>> Not sure if a blocker but worth tracking.

>>> Query 10 - Batch/Bounded
>>> Version  Runtime(sec)   Events(/sec)Results
>>>2.3.0   3.627609.1  1
>>>2.4.0  30.8 3244.3  1

>>> Query 10 - Streaming/Unbounded
>>> Version  Runtime(sec)   Events(/sec)Results
>>>2.3.0   6.315873.0  1
>>>2.4.0 101.1  989.4  1

>>> On Thu, Mar 8, 2018 at 8:54 AM, Romain Manni-Bucau
>>>  wrote:
>>> > -1:
>>> > a) still consider waitUntilFinish broken and a big blocker
>>> > b) restrictiontracker api changed and is not backward compatible
>>> > (
https://github.com/apache/beam/commit/e0034314ad196d2274cef9831ed63e090bf4d4c1#diff-098d7247eb1e9d9423bfa2ae2da38a9d
)
>>> >
>>> > with workarounds and fixes for these two issues the other parts
work (spark,
>>> > flink, direct runner, java core) on my projects
>>> >
>>> >
>>> >
>>> > Romain Manni-Bucau
>>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>> >
>>> > 2018-03-08 6:26 GMT+01:00 Robert Bradshaw :
>>> >>
>>> >> Hi everyone,
>>> >>
>>> >> Please review and vote on the release candidate #2 for the
version 2.4.0,
>>> >> as follows:
>>> >> [ ] +1, Approve the release
>>> >> [ ] -1, Do not approve the release (please provide specific
comments)
>>> >>
>>> >> The complete staging area is available for your review, which
includes:
>>> >> * JIRA release notes [1],
>>> >> * the official Apache source release to be deployed to
dist.apache.org
>>> >> [2],
>>> >> which is signed with the key with

Re: [VOTE] Release 2.4.0, release candidate #2

2018-03-09 Thread Robert Bradshaw

Also, I figured out how to create Python wheels and put them up at
https://test.pypi.org/project/apache-beam/#files . I'm in the process of
documenting and cleaning up this process.


On Wed, Mar 7, 2018 at 9:26 PM Robert Bradshaw  wrote:

> Hi everyone,
>
> Please review and vote on the release candidate #2 for the version 2.4.0,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org
> [2],
> which is signed with the key with fingerprint BDC9 89B0 1BD2 A463 6010
>A1CA 8F15 5E09 610D 69FB [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.4.0-RC2" [5],
> * website pull request listing the release and publishing the API reference
> manual [6].
> * Java artifacts were built with Maven 3.2.5 and OpenJDK 1.8.0_112.
> * Python artifact are deployed along with the source release to the
> dist.apache.org [2]. If I am able to figure out how to build the wheels, I
> will post them there as well.
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> - Robert
>
> [1]
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12342682=12319527
> [2] https://dist.apache.org/repos/dist/dev/beam/2.4.0/
> [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1030/
> [5] https://github.com/apache/beam/tree/v2.4.0-RC2
> [6] https://github.com/apache/beam-site/pull/398
>

Re: [VOTE] Release 2.4.0, release candidate #2

2018-03-09 Thread Robert Bradshaw

The symptoms were a Coder that did not properly implement structuralValue
(e.g. it returned the original object whose equals() was not overridden)
but that did have deterministic serialization. Here at every element we're
comparing serialized forms to ensure the elements were not mutated, which
is better than outright failure, but expensive.


On Fri, Mar 9, 2018 at 4:45 AM Reuven Lax  wrote:

> This makes me worry that the performance regression is elsewhere, in which
> case we should investigate.
>
>
> On Fri, Mar 9, 2018 at 1:02 AM Etienne Chauchot 
> wrote:
>
>> Le jeudi 08 mars 2018 à 20:01 +, Reuven Lax a écrit :
>>
>> Does Nexmark use SerializableCoder?
>>
>>
>> Actually SerializableCoder is registered in Nexmark but the default and
>> the current configuration are set to use "by hand serialization". See:
>>
>> public static void setupPipeline(CoderStrategy coderStrategy, Pipeline p) {
>>   CoderRegistry registry = p.getCoderRegistry();
>>   switch (coderStrategy) {
>> case HAND:
>>   registry.registerCoderForClass(Auction.class, Auction.CODER);
>>   registry.registerCoderForClass(AuctionBid.class, AuctionBid.CODER);
>>   registry.registerCoderForClass(AuctionCount.class, AuctionCount.CODER);
>>   registry.registerCoderForClass(AuctionPrice.class, AuctionPrice.CODER);
>>   registry.registerCoderForClass(Bid.class, Bid.CODER);
>>   registry.registerCoderForClass(CategoryPrice.class, 
>> CategoryPrice.CODER);
>>   registry.registerCoderForClass(Event.class, Event.CODER);
>>   registry.registerCoderForClass(IdNameReserve.class, 
>> IdNameReserve.CODER);
>>   registry.registerCoderForClass(NameCityStateId.class, 
>> NameCityStateId.CODER);
>>   registry.registerCoderForClass(Person.class, Person.CODER);
>>   registry.registerCoderForClass(SellerPrice.class, SellerPrice.CODER);
>>   registry.registerCoderForClass(Done.class, Done.CODER);
>>   registry.registerCoderForClass(BidsPerSession.class, 
>> BidsPerSession.CODER);
>>   break;
>> case AVRO:
>>   registry.registerCoderProvider(AvroCoder.getCoderProvider());
>>   break;
>> case JAVA:
>>   registry.registerCoderProvider(SerializableCoder.getCoderProvider());
>>   break;
>>   }
>> }
>>
>>
>> Etienne
>>
>>
>>
>> On Thu, Mar 8, 2018 at 10:42 AM Robert Bradshaw 
>> wrote:
>>
>> I put the validation checklist spreadsheet is up at
>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit?ts=5a1c7310#gid=1663314475
>>
>> Regarding the direct runner regression on query 10, this is
>> understandable given how mutation detection has been changed for
>> serializable coders (and should be tracked, probably fixed by avoiding
>> SerializableCoder). It should not affect other runners. Could you file a
>> bug?
>>
>> Regarding waitUntilFinish, this is a bug but not a blocker--it's been
>> this way since teardown was introduced. There are many nice-to-haves that
>> one could merge from master to the release branch, but we've seen where
>> that trend leads.
>>
>> Regarding the backwards incompatible changes in restriction tracker, this
>> is (as I understand it) a change to the experimental SDF API. Eugene, do
>> you want to comment on this?
>>
>>
>>
>> On Thu, Mar 8, 2018 at 2:07 AM Ismaël Mejía  wrote:
>>
>> I confirm that the new release fixes both problems reported previously:
>>
>> - python package name
>> - nexmark query 10 mutability issue with the direct runner.
>>
>> One extra regression is that the the fix produced a way longer
>> execution time on the query.
>> Not sure if a blocker but worth tracking.
>>
>> Query 10 - Batch/Bounded
>> Version  Runtime(sec)   Events(/sec)Results
>>   2.3.0   3.627609.1  1
>>   2.4.0  30.8 3244.3  1
>>
>> Query 10 - Streaming/Unbounded
>> Version  Runtime(sec)   Events(/sec)Results
>>   2.3.0   6.315873.0  1
>>   2.4.0 101.1  989.4  1
>>
>> On Thu, Mar 8, 2018 at 8:54 AM, Romain Manni-Bucau
>>  wrote:
>> > -1:
>> > a) still consider waitUntilFinish broken and a big blocker
>> > b) restrictiontracker api changed and is not backward compatible
>> > (
>> https://github.com/apache/beam/commit/e0034314ad196d2274cef9831ed63e090bf4d4c1#diff-098d7247eb1e9d9423bfa2ae2da38a9d
>> )
>> >
>> > with workarounds and fixes for these two issues the other parts work
>> (spark,
>> > flink, direct runner, java core) on my projects
>> >
>> >
>> >
>> > Romain Manni-Bucau
>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>> >
>> > 2018-03-08 6:26 GMT+01:00 Robert Bradshaw :
>> >>
>> >> Hi everyone,
>> >>
>> >> Please review and vote on the release candidate #2 for the version
>> 2.4.0,
>> >> as follows:
>> >> [ ] +1, Approve the release
>> >> [ ] -1, Do not approve the release (please

Re: [VOTE] Release 2.4.0, release candidate #2

2018-03-09 Thread Reuven Lax

This makes me worry that the performance regression is elsewhere, in which
case we should investigate.


On Fri, Mar 9, 2018 at 1:02 AM Etienne Chauchot 
wrote:

> Le jeudi 08 mars 2018 à 20:01 +, Reuven Lax a écrit :
>
> Does Nexmark use SerializableCoder?
>
>
> Actually SerializableCoder is registered in Nexmark but the default and
> the current configuration are set to use "by hand serialization". See:
>
> public static void setupPipeline(CoderStrategy coderStrategy, Pipeline p) {
>   CoderRegistry registry = p.getCoderRegistry();
>   switch (coderStrategy) {
> case HAND:
>   registry.registerCoderForClass(Auction.class, Auction.CODER);
>   registry.registerCoderForClass(AuctionBid.class, AuctionBid.CODER);
>   registry.registerCoderForClass(AuctionCount.class, AuctionCount.CODER);
>   registry.registerCoderForClass(AuctionPrice.class, AuctionPrice.CODER);
>   registry.registerCoderForClass(Bid.class, Bid.CODER);
>   registry.registerCoderForClass(CategoryPrice.class, 
> CategoryPrice.CODER);
>   registry.registerCoderForClass(Event.class, Event.CODER);
>   registry.registerCoderForClass(IdNameReserve.class, 
> IdNameReserve.CODER);
>   registry.registerCoderForClass(NameCityStateId.class, 
> NameCityStateId.CODER);
>   registry.registerCoderForClass(Person.class, Person.CODER);
>   registry.registerCoderForClass(SellerPrice.class, SellerPrice.CODER);
>   registry.registerCoderForClass(Done.class, Done.CODER);
>   registry.registerCoderForClass(BidsPerSession.class, 
> BidsPerSession.CODER);
>   break;
> case AVRO:
>   registry.registerCoderProvider(AvroCoder.getCoderProvider());
>   break;
> case JAVA:
>   registry.registerCoderProvider(SerializableCoder.getCoderProvider());
>   break;
>   }
> }
>
>
> Etienne
>
>
>
> On Thu, Mar 8, 2018 at 10:42 AM Robert Bradshaw 
> wrote:
>
> I put the validation checklist spreadsheet is up at
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit?ts=5a1c7310#gid=1663314475
>
> Regarding the direct runner regression on query 10, this is understandable
> given how mutation detection has been changed for serializable coders (and
> should be tracked, probably fixed by avoiding SerializableCoder). It should
> not affect other runners. Could you file a bug?
>
> Regarding waitUntilFinish, this is a bug but not a blocker--it's been
> this way since teardown was introduced. There are many nice-to-haves that
> one could merge from master to the release branch, but we've seen where
> that trend leads.
>
> Regarding the backwards incompatible changes in restriction tracker, this
> is (as I understand it) a change to the experimental SDF API. Eugene, do
> you want to comment on this?
>
>
>
> On Thu, Mar 8, 2018 at 2:07 AM Ismaël Mejía  wrote:
>
> I confirm that the new release fixes both problems reported previously:
>
> - python package name
> - nexmark query 10 mutability issue with the direct runner.
>
> One extra regression is that the the fix produced a way longer
> execution time on the query.
> Not sure if a blocker but worth tracking.
>
> Query 10 - Batch/Bounded
> Version  Runtime(sec)   Events(/sec)Results
>   2.3.0   3.627609.1  1
>   2.4.0  30.8 3244.3  1
>
> Query 10 - Streaming/Unbounded
> Version  Runtime(sec)   Events(/sec)Results
>   2.3.0   6.315873.0  1
>   2.4.0 101.1  989.4  1
>
> On Thu, Mar 8, 2018 at 8:54 AM, Romain Manni-Bucau
>  wrote:
> > -1:
> > a) still consider waitUntilFinish broken and a big blocker
> > b) restrictiontracker api changed and is not backward compatible
> > (
> https://github.com/apache/beam/commit/e0034314ad196d2274cef9831ed63e090bf4d4c1#diff-098d7247eb1e9d9423bfa2ae2da38a9d
> )
> >
> > with workarounds and fixes for these two issues the other parts work
> (spark,
> > flink, direct runner, java core) on my projects
> >
> >
> >
> > Romain Manni-Bucau
> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >
> > 2018-03-08 6:26 GMT+01:00 Robert Bradshaw :
> >>
> >> Hi everyone,
> >>
> >> Please review and vote on the release candidate #2 for the version
> 2.4.0,
> >> as follows:
> >> [ ] +1, Approve the release
> >> [ ] -1, Do not approve the release (please provide specific comments)
> >>
> >> The complete staging area is available for your review, which includes:
> >> * JIRA release notes [1],
> >> * the official Apache source release to be deployed to dist.apache.org
> >> [2],
> >> which is signed with the key with fingerprint BDC9 89B0 1BD2 A463 6010
> >>A1CA 8F15 5E09 610D 69FB [3],
> >> * all artifacts to be deployed to the Maven Central Repository [4],
> >> * source code tag "v2.4.0-RC2" [5],
> >> * website pull request listing the release and publishing the API
> >> reference
> >>

Re: [VOTE] Release 2.4.0, release candidate #2

2018-03-09 Thread Etienne Chauchot

Le jeudi 08 mars 2018 à 20:01 +, Reuven Lax a écrit :
> Does Nexmark use SerializableCoder?
Actually SerializableCoder is registered in Nexmark but the default and the 
current configuration are set to use "by
hand serialization". See:
public static void setupPipeline(CoderStrategy coderStrategy, Pipeline p) {
  CoderRegistry registry = p.getCoderRegistry();  switch (coderStrategy) {
case HAND:
  registry.registerCoderForClass(Auction.class, Auction.CODER);  
registry.registerCoderForClass(AuctionBid.class, AuctionBid.CODER);  
registry.registerCoderForClass(AuctionCount.class, AuctionCount.CODER);  
registry.registerCoderForClass(AuctionPrice.class, AuctionPrice.CODER);  
registry.registerCoderForClass(Bid.class, Bid.CODER);  
registry.registerCoderForClass(CategoryPrice.class, CategoryPrice.CODER);  
registry.registerCoderForClass(Event.class, Event.CODER);  
registry.registerCoderForClass(IdNameReserve.class, IdNameReserve.CODER);  
registry.registerCoderForClass(NameCityStateId.class, NameCityStateId.CODER);   
   registry.registerCoderForClass(Person.class, Person.CODER);  
registry.registerCoderForClass(SellerPrice.class, SellerPrice.CODER);  
registry.registerCoderForClass(Done.class, Done.CODER);  
registry.registerCoderForClass(BidsPerSession.class, BidsPerSession.CODER); 
 break;case AVRO:
  registry.registerCoderProvider(AvroCoder.getCoderProvider());  break; 
   case JAVA:
  registry.registerCoderProvider(SerializableCoder.getCoderProvider()); 
 break;  }
}
Etienne
> 

> On Thu, Mar 8, 2018 at 10:42 AM Robert Bradshaw  wrote:> 
> > I put the validation checklist spreadsheet is up at 
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit?ts=5a1c7310#gid=1663314475>
>  > > > Regarding the direct runner regression on query 10, this is 
> understandable given how mutation detection has been changed for serializable 
> coders (and should be tracked, probably fixed by avoiding SerializableCoder). 
> It should not affect other runners. Could you file a bug? 
> > > > Regarding > > waitUntilFinish, this is a bug but not a blocker--it's 
> > > > been this way since teardown was introduced. There are many 
> > > > nice-to-haves that one could merge from master to the release branch, 
> > > > but we've seen where that trend leads. 

> > Regarding the backwards incompatible changes in restriction tracker, this 
> > is (as I understand it) a change to the experimental SDF API. Eugene, do 
> > you want to comment on this? 

> > 

> > On Thu, Mar 8, 2018 at 2:07 AM Ismaël Mejía  wrote:> > > 
> > I confirm that the new release fixes both problems reported previously:
> > > 

> > > 
> > > - python package name
> > > 
> > > - nexmark query 10 mutability issue with the direct runner.
> > > 

> > > 
> > > One extra regression is that the the fix produced a way longer
> > > 
> > > execution time on the query.
> > > 
> > > Not sure if a blocker but worth tracking.
> > > 

> > > 
> > > Query 10 - Batch/Bounded
> > > 
> > > Version  Runtime(sec)   Events(/sec)    Results
> > > 
> > >   2.3.0           3.6        27609.1          1
> > > 
> > >   2.4.0          30.8         3244.3          1
> > > 

> > > 
> > > Query 10 - Streaming/Unbounded
> > > 
> > > Version  Runtime(sec)   Events(/sec)    Results
> > > 
> > >   2.3.0           6.3        15873.0          1
> > > 
> > >   2.4.0         101.1          989.4          1
> > > 

> > > 
> > > On Thu, Mar 8, 2018 at 8:54 AM, Romain Manni-Bucau
> > > 
> > >  wrote:
> > > 
> > > > -1:
> > > 
> > > > a) still consider waitUntilFinish broken and a big blocker
> > > 
> > > > b) restrictiontracker api changed and is not backward compatible
> > > 
> > > > (https://github.com/apache/beam/commit/e0034314ad196d2274cef9831ed63e090bf4d4c1#diff-098d7247eb1e9d9423bfa2ae2da38a9d)
> > > 
> > > >
> > > 
> > > > with workarounds and fixes for these two issues the other parts work 
> > > > (spark,
> > > 
> > > > flink, direct runner, java core) on my projects
> > > 
> > > >
> > > 
> > > >
> > > 
> > > >
> > > 
> > > > Romain Manni-Bucau
> > > 
> > > > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> > > 
> > > >
> > > 
> > > > 2018-03-08 6:26 GMT+01:00 Robert Bradshaw :
> > > 
> > > >>
> > > 
> > > >> Hi everyone,
> > > 
> > > >>
> > > 
> > > >> Please review and vote on the release candidate #2 for the version 
> > > >> 2.4.0,
> > > 
> > > >> as follows:
> > > 
> > > >> [ ] +1, Approve the release
> > > 
> > > >> [ ] -1, Do not approve the release (please provide specific comments)
> > > 
> > > >>
> > > 
> > > >> The complete staging area is available for your review, which includes:
> > > 
> > > >> * JIRA release notes [1],
> > > 
> > > >> * the official Apache source release to be deployed to dist.apache.org
> > > 
> > > >> [2],
> > > 
> > > >> which is signed with the key

Re: [VOTE] Release 2.4.0, release candidate #2

2018-03-08 Thread Lukasz Cwik

Yes, the release guide has a segment "Update release specific
configurations" that has a tidbit about this.

On Thu, Mar 8, 2018 at 3:45 PM, Alan Myrvold  wrote:

> The dataflow java worker version wasn't updated on the branch as in past
> releases ... should it be?
> https://issues.apache.org/jira/browse/BEAM-3815
>
>
> On Thu, Mar 8, 2018 at 1:40 PM Romain Manni-Bucau 
> wrote:
>
>> Can still be provided as a generic one (like the an offset or key based
>> one) but good enough for now, right, was just surprising to not see it when
>> checking the breakage.
>>
>> Le 8 mars 2018 22:05, "Eugene Kirpichov"  a écrit :
>>
>> All SDF-related method annotations in DoFn are marked @Experimental. I
>> guess that should apply to RestrictionTracker too, but I wouldn't be too
>> worried about that, since it only makes sense to use in the context of
>> those methods.
>>
>> On Thu, Mar 8, 2018 at 12:36 PM Romain Manni-Bucau 
>> wrote:
>>
>>> Hmm, does sdf api misses some @Experimental then?
>>>
>>> To clarify: for waitUntilFinish I'm fine with the 2.4 as this but cant
>>> +1 or +0 since none of my tests pass reliably in current state without a
>>> retry strategy making the call useless.
>>>
>>> Le 8 mars 2018 21:02, "Reuven Lax"  a écrit :
>>>
 Does Nexmark use SerializableCoder?


 On Thu, Mar 8, 2018 at 10:42 AM Robert Bradshaw 
 wrote:

> I put the validation checklist spreadsheet is up at
> https://docs.google.com/spreadsheets/d/1qk-
> N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit?ts=5a1c7310#
> gid=1663314475
>
> Regarding the direct runner regression on query 10, this is
> understandable given how mutation detection has been changed for
> serializable coders (and should be tracked, probably fixed by avoiding
> SerializableCoder). It should not affect other runners. Could you file a
> bug?
>
> Regarding waitUntilFinish, this is a bug but not a blocker--it's been
> this way since teardown was introduced. There are many nice-to-haves that
> one could merge from master to the release branch, but we've seen where
> that trend leads.
>
> Regarding the backwards incompatible changes in restriction tracker,
> this is (as I understand it) a change to the experimental SDF API. Eugene,
> do you want to comment on this?
>
>
>
> On Thu, Mar 8, 2018 at 2:07 AM Ismaël Mejía  wrote:
>
>> I confirm that the new release fixes both problems reported
>> previously:
>>
>> - python package name
>> - nexmark query 10 mutability issue with the direct runner.
>>
>> One extra regression is that the the fix produced a way longer
>> execution time on the query.
>> Not sure if a blocker but worth tracking.
>>
>> Query 10 - Batch/Bounded
>> Version  Runtime(sec)   Events(/sec)Results
>>   2.3.0   3.627609.1  1
>>   2.4.0  30.8 3244.3  1
>>
>> Query 10 - Streaming/Unbounded
>> Version  Runtime(sec)   Events(/sec)Results
>>   2.3.0   6.315873.0  1
>>   2.4.0 101.1  989.4  1
>>
>> On Thu, Mar 8, 2018 at 8:54 AM, Romain Manni-Bucau
>>  wrote:
>> > -1:
>> > a) still consider waitUntilFinish broken and a big blocker
>> > b) restrictiontracker api changed and is not backward compatible
>> > (https://github.com/apache/beam/commit/
>> e0034314ad196d2274cef9831ed63e090bf4d4c1#diff-
>> 098d7247eb1e9d9423bfa2ae2da38a9d)
>> >
>> > with workarounds and fixes for these two issues the other parts
>> work (spark,
>> > flink, direct runner, java core) on my projects
>> >
>> >
>> >
>> > Romain Manni-Bucau
>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>> >
>> > 2018-03-08 6:26 GMT+01:00 Robert Bradshaw :
>> >>
>> >> Hi everyone,
>> >>
>> >> Please review and vote on the release candidate #2 for the version
>> 2.4.0,
>> >> as follows:
>> >> [ ] +1, Approve the release
>> >> [ ] -1, Do not approve the release (please provide specific
>> comments)
>> >>
>> >> The complete staging area is available for your review, which
>> includes:
>> >> * JIRA release notes [1],
>> >> * the official Apache source release to be deployed to
>> dist.apache.org
>> >> [2],
>> >> which is signed with the key with fingerprint BDC9 89B0 1BD2 A463
>> 6010
>> >>A1CA 8F15 5E09 610D 69FB [3],
>> >> * all artifacts to be deployed to the Maven Central Repository [4],
>> >> * source code tag "v2.4.0-RC2" [5],
>> >> * website pull request listing the release and publishing the API
>> >> reference
>> >> manual [6].

Re: [VOTE] Release 2.4.0, release candidate #2

2018-03-08 Thread Alan Myrvold

The dataflow java worker version wasn't updated on the branch as in past
releases ... should it be?
https://issues.apache.org/jira/browse/BEAM-3815


On Thu, Mar 8, 2018 at 1:40 PM Romain Manni-Bucau 
wrote:

> Can still be provided as a generic one (like the an offset or key based
> one) but good enough for now, right, was just surprising to not see it when
> checking the breakage.
>
> Le 8 mars 2018 22:05, "Eugene Kirpichov"  a écrit :
>
> All SDF-related method annotations in DoFn are marked @Experimental. I
> guess that should apply to RestrictionTracker too, but I wouldn't be too
> worried about that, since it only makes sense to use in the context of
> those methods.
>
> On Thu, Mar 8, 2018 at 12:36 PM Romain Manni-Bucau 
> wrote:
>
>> Hmm, does sdf api misses some @Experimental then?
>>
>> To clarify: for waitUntilFinish I'm fine with the 2.4 as this but cant +1
>> or +0 since none of my tests pass reliably in current state without a retry
>> strategy making the call useless.
>>
>> Le 8 mars 2018 21:02, "Reuven Lax"  a écrit :
>>
>>> Does Nexmark use SerializableCoder?
>>>
>>>
>>> On Thu, Mar 8, 2018 at 10:42 AM Robert Bradshaw 
>>> wrote:
>>>
 I put the validation checklist spreadsheet is up at
 https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit?ts=5a1c7310#gid=1663314475

 Regarding the direct runner regression on query 10, this is
 understandable given how mutation detection has been changed for
 serializable coders (and should be tracked, probably fixed by avoiding
 SerializableCoder). It should not affect other runners. Could you file a
 bug?

 Regarding waitUntilFinish, this is a bug but not a blocker--it's been
 this way since teardown was introduced. There are many nice-to-haves that
 one could merge from master to the release branch, but we've seen where
 that trend leads.

 Regarding the backwards incompatible changes in restriction tracker,
 this is (as I understand it) a change to the experimental SDF API. Eugene,
 do you want to comment on this?



 On Thu, Mar 8, 2018 at 2:07 AM Ismaël Mejía  wrote:

> I confirm that the new release fixes both problems reported previously:
>
> - python package name
> - nexmark query 10 mutability issue with the direct runner.
>
> One extra regression is that the the fix produced a way longer
> execution time on the query.
> Not sure if a blocker but worth tracking.
>
> Query 10 - Batch/Bounded
> Version  Runtime(sec)   Events(/sec)Results
>   2.3.0   3.627609.1  1
>   2.4.0  30.8 3244.3  1
>
> Query 10 - Streaming/Unbounded
> Version  Runtime(sec)   Events(/sec)Results
>   2.3.0   6.315873.0  1
>   2.4.0 101.1  989.4  1
>
> On Thu, Mar 8, 2018 at 8:54 AM, Romain Manni-Bucau
>  wrote:
> > -1:
> > a) still consider waitUntilFinish broken and a big blocker
> > b) restrictiontracker api changed and is not backward compatible
> > (
> https://github.com/apache/beam/commit/e0034314ad196d2274cef9831ed63e090bf4d4c1#diff-098d7247eb1e9d9423bfa2ae2da38a9d
> )
> >
> > with workarounds and fixes for these two issues the other parts work
> (spark,
> > flink, direct runner, java core) on my projects
> >
> >
> >
> > Romain Manni-Bucau
> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >
> > 2018-03-08 6:26 GMT+01:00 Robert Bradshaw :
> >>
> >> Hi everyone,
> >>
> >> Please review and vote on the release candidate #2 for the version
> 2.4.0,
> >> as follows:
> >> [ ] +1, Approve the release
> >> [ ] -1, Do not approve the release (please provide specific
> comments)
> >>
> >> The complete staging area is available for your review, which
> includes:
> >> * JIRA release notes [1],
> >> * the official Apache source release to be deployed to
> dist.apache.org
> >> [2],
> >> which is signed with the key with fingerprint BDC9 89B0 1BD2 A463
> 6010
> >>A1CA 8F15 5E09 610D 69FB [3],
> >> * all artifacts to be deployed to the Maven Central Repository [4],
> >> * source code tag "v2.4.0-RC2" [5],
> >> * website pull request listing the release and publishing the API
> >> reference
> >> manual [6].
> >> * Java artifacts were built with Maven 3.2.5 and OpenJDK 1.8.0_112.
> >> * Python artifact are deployed along with the source release to the
> >> dist.apache.org [2]. If I am able to figure out how to build the
> wheels, I
> >> will post them there as well.
> >>
> >> The vote will be open for at

Re: [VOTE] Release 2.4.0, release candidate #2

2018-03-08 Thread Romain Manni-Bucau

Can still be provided as a generic one (like the an offset or key based
one) but good enough for now, right, was just surprising to not see it when
checking the breakage.

Le 8 mars 2018 22:05, "Eugene Kirpichov"  a écrit :

All SDF-related method annotations in DoFn are marked @Experimental. I
guess that should apply to RestrictionTracker too, but I wouldn't be too
worried about that, since it only makes sense to use in the context of
those methods.

On Thu, Mar 8, 2018 at 12:36 PM Romain Manni-Bucau 
wrote:

> Hmm, does sdf api misses some @Experimental then?
>
> To clarify: for waitUntilFinish I'm fine with the 2.4 as this but cant +1
> or +0 since none of my tests pass reliably in current state without a retry
> strategy making the call useless.
>
> Le 8 mars 2018 21:02, "Reuven Lax"  a écrit :
>
>> Does Nexmark use SerializableCoder?
>>
>>
>> On Thu, Mar 8, 2018 at 10:42 AM Robert Bradshaw 
>> wrote:
>>
>>> I put the validation checklist spreadsheet is up at
>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-
>>> oLFo_ZXBpJw/edit?ts=5a1c7310#gid=1663314475
>>>
>>> Regarding the direct runner regression on query 10, this is
>>> understandable given how mutation detection has been changed for
>>> serializable coders (and should be tracked, probably fixed by avoiding
>>> SerializableCoder). It should not affect other runners. Could you file a
>>> bug?
>>>
>>> Regarding waitUntilFinish, this is a bug but not a blocker--it's been
>>> this way since teardown was introduced. There are many nice-to-haves that
>>> one could merge from master to the release branch, but we've seen where
>>> that trend leads.
>>>
>>> Regarding the backwards incompatible changes in restriction tracker,
>>> this is (as I understand it) a change to the experimental SDF API. Eugene,
>>> do you want to comment on this?
>>>
>>>
>>>
>>> On Thu, Mar 8, 2018 at 2:07 AM Ismaël Mejía  wrote:
>>>
 I confirm that the new release fixes both problems reported previously:

 - python package name
 - nexmark query 10 mutability issue with the direct runner.

 One extra regression is that the the fix produced a way longer
 execution time on the query.
 Not sure if a blocker but worth tracking.

 Query 10 - Batch/Bounded
 Version  Runtime(sec)   Events(/sec)Results
   2.3.0   3.627609.1  1
   2.4.0  30.8 3244.3  1

 Query 10 - Streaming/Unbounded
 Version  Runtime(sec)   Events(/sec)Results
   2.3.0   6.315873.0  1
   2.4.0 101.1  989.4  1

 On Thu, Mar 8, 2018 at 8:54 AM, Romain Manni-Bucau
  wrote:
 > -1:
 > a) still consider waitUntilFinish broken and a big blocker
 > b) restrictiontracker api changed and is not backward compatible
 > (https://github.com/apache/beam/commit/e0034314ad196d2274cef9831ed63e
 090bf4d4c1#diff-098d7247eb1e9d9423bfa2ae2da38a9d)
 >
 > with workarounds and fixes for these two issues the other parts work
 (spark,
 > flink, direct runner, java core) on my projects
 >
 >
 >
 > Romain Manni-Bucau
 > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
 >
 > 2018-03-08 6:26 GMT+01:00 Robert Bradshaw :
 >>
 >> Hi everyone,
 >>
 >> Please review and vote on the release candidate #2 for the version
 2.4.0,
 >> as follows:
 >> [ ] +1, Approve the release
 >> [ ] -1, Do not approve the release (please provide specific comments)
 >>
 >> The complete staging area is available for your review, which
 includes:
 >> * JIRA release notes [1],
 >> * the official Apache source release to be deployed to
 dist.apache.org
 >> [2],
 >> which is signed with the key with fingerprint BDC9 89B0 1BD2 A463
 6010
 >>A1CA 8F15 5E09 610D 69FB [3],
 >> * all artifacts to be deployed to the Maven Central Repository [4],
 >> * source code tag "v2.4.0-RC2" [5],
 >> * website pull request listing the release and publishing the API
 >> reference
 >> manual [6].
 >> * Java artifacts were built with Maven 3.2.5 and OpenJDK 1.8.0_112.
 >> * Python artifact are deployed along with the source release to the
 >> dist.apache.org [2]. If I am able to figure out how to build the
 wheels, I
 >> will post them there as well.
 >>
 >> The vote will be open for at least 72 hours. It is adopted by
 majority
 >> approval, with at least 3 PMC affirmative votes.
 >>
 >> Thanks,
 >> - Robert
 >>
 >> [1]
 >>
 >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
 version=12342682=12319527
 >> [2] https://dist.apache.org/repos/dist/dev/beam/2.4.0/
 >> [3]

Re: [VOTE] Release 2.4.0, release candidate #2

2018-03-08 Thread Eugene Kirpichov

All SDF-related method annotations in DoFn are marked @Experimental. I
guess that should apply to RestrictionTracker too, but I wouldn't be too
worried about that, since it only makes sense to use in the context of
those methods.

On Thu, Mar 8, 2018 at 12:36 PM Romain Manni-Bucau 
wrote:

> Hmm, does sdf api misses some @Experimental then?
>
> To clarify: for waitUntilFinish I'm fine with the 2.4 as this but cant +1
> or +0 since none of my tests pass reliably in current state without a retry
> strategy making the call useless.
>
> Le 8 mars 2018 21:02, "Reuven Lax"  a écrit :
>
>> Does Nexmark use SerializableCoder?
>>
>>
>> On Thu, Mar 8, 2018 at 10:42 AM Robert Bradshaw 
>> wrote:
>>
>>> I put the validation checklist spreadsheet is up at
>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit?ts=5a1c7310#gid=1663314475
>>>
>>> Regarding the direct runner regression on query 10, this is
>>> understandable given how mutation detection has been changed for
>>> serializable coders (and should be tracked, probably fixed by avoiding
>>> SerializableCoder). It should not affect other runners. Could you file a
>>> bug?
>>>
>>> Regarding waitUntilFinish, this is a bug but not a blocker--it's been
>>> this way since teardown was introduced. There are many nice-to-haves that
>>> one could merge from master to the release branch, but we've seen where
>>> that trend leads.
>>>
>>> Regarding the backwards incompatible changes in restriction tracker,
>>> this is (as I understand it) a change to the experimental SDF API. Eugene,
>>> do you want to comment on this?
>>>
>>>
>>>
>>> On Thu, Mar 8, 2018 at 2:07 AM Ismaël Mejía  wrote:
>>>
 I confirm that the new release fixes both problems reported previously:

 - python package name
 - nexmark query 10 mutability issue with the direct runner.

 One extra regression is that the the fix produced a way longer
 execution time on the query.
 Not sure if a blocker but worth tracking.

 Query 10 - Batch/Bounded
 Version  Runtime(sec)   Events(/sec)Results
   2.3.0   3.627609.1  1
   2.4.0  30.8 3244.3  1

 Query 10 - Streaming/Unbounded
 Version  Runtime(sec)   Events(/sec)Results
   2.3.0   6.315873.0  1
   2.4.0 101.1  989.4  1

 On Thu, Mar 8, 2018 at 8:54 AM, Romain Manni-Bucau
  wrote:
 > -1:
 > a) still consider waitUntilFinish broken and a big blocker
 > b) restrictiontracker api changed and is not backward compatible
 > (
 https://github.com/apache/beam/commit/e0034314ad196d2274cef9831ed63e090bf4d4c1#diff-098d7247eb1e9d9423bfa2ae2da38a9d
 )
 >
 > with workarounds and fixes for these two issues the other parts work
 (spark,
 > flink, direct runner, java core) on my projects
 >
 >
 >
 > Romain Manni-Bucau
 > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
 >
 > 2018-03-08 6:26 GMT+01:00 Robert Bradshaw :
 >>
 >> Hi everyone,
 >>
 >> Please review and vote on the release candidate #2 for the version
 2.4.0,
 >> as follows:
 >> [ ] +1, Approve the release
 >> [ ] -1, Do not approve the release (please provide specific comments)
 >>
 >> The complete staging area is available for your review, which
 includes:
 >> * JIRA release notes [1],
 >> * the official Apache source release to be deployed to
 dist.apache.org
 >> [2],
 >> which is signed with the key with fingerprint BDC9 89B0 1BD2 A463
 6010
 >>A1CA 8F15 5E09 610D 69FB [3],
 >> * all artifacts to be deployed to the Maven Central Repository [4],
 >> * source code tag "v2.4.0-RC2" [5],
 >> * website pull request listing the release and publishing the API
 >> reference
 >> manual [6].
 >> * Java artifacts were built with Maven 3.2.5 and OpenJDK 1.8.0_112.
 >> * Python artifact are deployed along with the source release to the
 >> dist.apache.org [2]. If I am able to figure out how to build the
 wheels, I
 >> will post them there as well.
 >>
 >> The vote will be open for at least 72 hours. It is adopted by
 majority
 >> approval, with at least 3 PMC affirmative votes.
 >>
 >> Thanks,
 >> - Robert
 >>
 >> [1]
 >>
 >>
 https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12342682=12319527
 >> [2] https://dist.apache.org/repos/dist/dev/beam/2.4.0/
 >> [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
 >> [4]
 https://repository.apache.org/content/repositories/orgapachebeam-1030/
 >> [5] https://github.com/apache/beam/tree/v2.4.0-RC2
 >> [6] https://github.com/apache/beam-site/pull/398
 >
 >

>>>

Re: [VOTE] Release 2.4.0, release candidate #2

2018-03-08 Thread Romain Manni-Bucau

Hmm, does sdf api misses some @Experimental then?

To clarify: for waitUntilFinish I'm fine with the 2.4 as this but cant +1
or +0 since none of my tests pass reliably in current state without a retry
strategy making the call useless.

Le 8 mars 2018 21:02, "Reuven Lax"  a écrit :

> Does Nexmark use SerializableCoder?
>
>
> On Thu, Mar 8, 2018 at 10:42 AM Robert Bradshaw 
> wrote:
>
>> I put the validation checklist spreadsheet is up at
>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-
>> oLFo_ZXBpJw/edit?ts=5a1c7310#gid=1663314475
>>
>> Regarding the direct runner regression on query 10, this is
>> understandable given how mutation detection has been changed for
>> serializable coders (and should be tracked, probably fixed by avoiding
>> SerializableCoder). It should not affect other runners. Could you file a
>> bug?
>>
>> Regarding waitUntilFinish, this is a bug but not a blocker--it's been
>> this way since teardown was introduced. There are many nice-to-haves that
>> one could merge from master to the release branch, but we've seen where
>> that trend leads.
>>
>> Regarding the backwards incompatible changes in restriction tracker, this
>> is (as I understand it) a change to the experimental SDF API. Eugene, do
>> you want to comment on this?
>>
>>
>>
>> On Thu, Mar 8, 2018 at 2:07 AM Ismaël Mejía  wrote:
>>
>>> I confirm that the new release fixes both problems reported previously:
>>>
>>> - python package name
>>> - nexmark query 10 mutability issue with the direct runner.
>>>
>>> One extra regression is that the the fix produced a way longer
>>> execution time on the query.
>>> Not sure if a blocker but worth tracking.
>>>
>>> Query 10 - Batch/Bounded
>>> Version  Runtime(sec)   Events(/sec)Results
>>>   2.3.0   3.627609.1  1
>>>   2.4.0  30.8 3244.3  1
>>>
>>> Query 10 - Streaming/Unbounded
>>> Version  Runtime(sec)   Events(/sec)Results
>>>   2.3.0   6.315873.0  1
>>>   2.4.0 101.1  989.4  1
>>>
>>> On Thu, Mar 8, 2018 at 8:54 AM, Romain Manni-Bucau
>>>  wrote:
>>> > -1:
>>> > a) still consider waitUntilFinish broken and a big blocker
>>> > b) restrictiontracker api changed and is not backward compatible
>>> > (https://github.com/apache/beam/commit/e0034314ad196d2274cef9831ed63e
>>> 090bf4d4c1#diff-098d7247eb1e9d9423bfa2ae2da38a9d)
>>> >
>>> > with workarounds and fixes for these two issues the other parts work
>>> (spark,
>>> > flink, direct runner, java core) on my projects
>>> >
>>> >
>>> >
>>> > Romain Manni-Bucau
>>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>> >
>>> > 2018-03-08 6:26 GMT+01:00 Robert Bradshaw :
>>> >>
>>> >> Hi everyone,
>>> >>
>>> >> Please review and vote on the release candidate #2 for the version
>>> 2.4.0,
>>> >> as follows:
>>> >> [ ] +1, Approve the release
>>> >> [ ] -1, Do not approve the release (please provide specific comments)
>>> >>
>>> >> The complete staging area is available for your review, which
>>> includes:
>>> >> * JIRA release notes [1],
>>> >> * the official Apache source release to be deployed to
>>> dist.apache.org
>>> >> [2],
>>> >> which is signed with the key with fingerprint BDC9 89B0 1BD2 A463 6010
>>> >>A1CA 8F15 5E09 610D 69FB [3],
>>> >> * all artifacts to be deployed to the Maven Central Repository [4],
>>> >> * source code tag "v2.4.0-RC2" [5],
>>> >> * website pull request listing the release and publishing the API
>>> >> reference
>>> >> manual [6].
>>> >> * Java artifacts were built with Maven 3.2.5 and OpenJDK 1.8.0_112.
>>> >> * Python artifact are deployed along with the source release to the
>>> >> dist.apache.org [2]. If I am able to figure out how to build the
>>> wheels, I
>>> >> will post them there as well.
>>> >>
>>> >> The vote will be open for at least 72 hours. It is adopted by majority
>>> >> approval, with at least 3 PMC affirmative votes.
>>> >>
>>> >> Thanks,
>>> >> - Robert
>>> >>
>>> >> [1]
>>> >>
>>> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>>> version=12342682=12319527
>>> >> [2] https://dist.apache.org/repos/dist/dev/beam/2.4.0/
>>> >> [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
>>> >> [4] https://repository.apache.org/content/repositories/
>>> orgapachebeam-1030/
>>> >> [5] https://github.com/apache/beam/tree/v2.4.0-RC2
>>> >> [6] https://github.com/apache/beam-site/pull/398
>>> >
>>> >
>>>
>>

Re: [VOTE] Release 2.4.0, release candidate #2

2018-03-08 Thread Robert Bradshaw

On Thu, Mar 8, 2018 at 12:02 PM Reuven Lax  wrote:

> Does Nexmark use SerializableCoder?
>

That's what the errors (and fix) for RC1 seemed to indicate.


> On Thu, Mar 8, 2018 at 10:42 AM Robert Bradshaw 
> wrote:
>
>> I put the validation checklist spreadsheet is up at
>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit?ts=5a1c7310#gid=1663314475
>>
>> Regarding the direct runner regression on query 10, this is
>> understandable given how mutation detection has been changed for
>> serializable coders (and should be tracked, probably fixed by avoiding
>> SerializableCoder). It should not affect other runners. Could you file a
>> bug?
>>
>> Regarding waitUntilFinish, this is a bug but not a blocker--it's been
>> this way since teardown was introduced. There are many nice-to-haves that
>> one could merge from master to the release branch, but we've seen where
>> that trend leads.
>>
>> Regarding the backwards incompatible changes in restriction tracker, this
>> is (as I understand it) a change to the experimental SDF API. Eugene, do
>> you want to comment on this?
>>
>>
>>
>> On Thu, Mar 8, 2018 at 2:07 AM Ismaël Mejía  wrote:
>>
>>> I confirm that the new release fixes both problems reported previously:
>>>
>>> - python package name
>>> - nexmark query 10 mutability issue with the direct runner.
>>>
>>> One extra regression is that the the fix produced a way longer
>>> execution time on the query.
>>> Not sure if a blocker but worth tracking.
>>>
>>> Query 10 - Batch/Bounded
>>> Version  Runtime(sec)   Events(/sec)Results
>>>   2.3.0   3.627609.1  1
>>>   2.4.0  30.8 3244.3  1
>>>
>>> Query 10 - Streaming/Unbounded
>>> Version  Runtime(sec)   Events(/sec)Results
>>>   2.3.0   6.315873.0  1
>>>   2.4.0 101.1  989.4  1
>>>
>>> On Thu, Mar 8, 2018 at 8:54 AM, Romain Manni-Bucau
>>>  wrote:
>>> > -1:
>>> > a) still consider waitUntilFinish broken and a big blocker
>>> > b) restrictiontracker api changed and is not backward compatible
>>> > (
>>> https://github.com/apache/beam/commit/e0034314ad196d2274cef9831ed63e090bf4d4c1#diff-098d7247eb1e9d9423bfa2ae2da38a9d
>>> )
>>> >
>>> > with workarounds and fixes for these two issues the other parts work
>>> (spark,
>>> > flink, direct runner, java core) on my projects
>>> >
>>> >
>>> >
>>> > Romain Manni-Bucau
>>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>> >
>>> > 2018-03-08 6:26 GMT+01:00 Robert Bradshaw :
>>> >>
>>> >> Hi everyone,
>>> >>
>>> >> Please review and vote on the release candidate #2 for the version
>>> 2.4.0,
>>> >> as follows:
>>> >> [ ] +1, Approve the release
>>> >> [ ] -1, Do not approve the release (please provide specific comments)
>>> >>
>>> >> The complete staging area is available for your review, which
>>> includes:
>>> >> * JIRA release notes [1],
>>> >> * the official Apache source release to be deployed to
>>> dist.apache.org
>>> >> [2],
>>> >> which is signed with the key with fingerprint BDC9 89B0 1BD2 A463 6010
>>> >>A1CA 8F15 5E09 610D 69FB [3],
>>> >> * all artifacts to be deployed to the Maven Central Repository [4],
>>> >> * source code tag "v2.4.0-RC2" [5],
>>> >> * website pull request listing the release and publishing the API
>>> >> reference
>>> >> manual [6].
>>> >> * Java artifacts were built with Maven 3.2.5 and OpenJDK 1.8.0_112.
>>> >> * Python artifact are deployed along with the source release to the
>>> >> dist.apache.org [2]. If I am able to figure out how to build the
>>> wheels, I
>>> >> will post them there as well.
>>> >>
>>> >> The vote will be open for at least 72 hours. It is adopted by majority
>>> >> approval, with at least 3 PMC affirmative votes.
>>> >>
>>> >> Thanks,
>>> >> - Robert
>>> >>
>>> >> [1]
>>> >>
>>> >>
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12342682=12319527
>>> >> [2] https://dist.apache.org/repos/dist/dev/beam/2.4.0/
>>> >> [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
>>> >> [4]
>>> https://repository.apache.org/content/repositories/orgapachebeam-1030/
>>> >> [5] https://github.com/apache/beam/tree/v2.4.0-RC2
>>> >> [6] https://github.com/apache/beam-site/pull/398
>>> >
>>> >
>>>
>>

Re: [VOTE] Release 2.4.0, release candidate #2

2018-03-08 Thread Eugene Kirpichov

Yes, SDF is an experimental API at this point, so backwards incompatible
changes are allowed and expected.

On Thu, Mar 8, 2018, 10:42 AM Robert Bradshaw  wrote:

> I put the validation checklist spreadsheet is up at
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit?ts=5a1c7310#gid=1663314475
>
> Regarding the direct runner regression on query 10, this is understandable
> given how mutation detection has been changed for serializable coders (and
> should be tracked, probably fixed by avoiding SerializableCoder). It should
> not affect other runners. Could you file a bug?
>
> Regarding waitUntilFinish, this is a bug but not a blocker--it's been
> this way since teardown was introduced. There are many nice-to-haves that
> one could merge from master to the release branch, but we've seen where
> that trend leads.
>
> Regarding the backwards incompatible changes in restriction tracker, this
> is (as I understand it) a change to the experimental SDF API. Eugene, do
> you want to comment on this?
>
>
>
> On Thu, Mar 8, 2018 at 2:07 AM Ismaël Mejía  wrote:
>
>> I confirm that the new release fixes both problems reported previously:
>>
>> - python package name
>> - nexmark query 10 mutability issue with the direct runner.
>>
>> One extra regression is that the the fix produced a way longer
>> execution time on the query.
>> Not sure if a blocker but worth tracking.
>>
>> Query 10 - Batch/Bounded
>> Version  Runtime(sec)   Events(/sec)Results
>>   2.3.0   3.627609.1  1
>>   2.4.0  30.8 3244.3  1
>>
>> Query 10 - Streaming/Unbounded
>> Version  Runtime(sec)   Events(/sec)Results
>>   2.3.0   6.315873.0  1
>>   2.4.0 101.1  989.4  1
>>
>> On Thu, Mar 8, 2018 at 8:54 AM, Romain Manni-Bucau
>>  wrote:
>> > -1:
>> > a) still consider waitUntilFinish broken and a big blocker
>> > b) restrictiontracker api changed and is not backward compatible
>> > (
>> https://github.com/apache/beam/commit/e0034314ad196d2274cef9831ed63e090bf4d4c1#diff-098d7247eb1e9d9423bfa2ae2da38a9d
>> )
>> >
>> > with workarounds and fixes for these two issues the other parts work
>> (spark,
>> > flink, direct runner, java core) on my projects
>> >
>> >
>> >
>> > Romain Manni-Bucau
>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>> >
>> > 2018-03-08 6:26 GMT+01:00 Robert Bradshaw :
>> >>
>> >> Hi everyone,
>> >>
>> >> Please review and vote on the release candidate #2 for the version
>> 2.4.0,
>> >> as follows:
>> >> [ ] +1, Approve the release
>> >> [ ] -1, Do not approve the release (please provide specific comments)
>> >>
>> >> The complete staging area is available for your review, which includes:
>> >> * JIRA release notes [1],
>> >> * the official Apache source release to be deployed to dist.apache.org
>> >> [2],
>> >> which is signed with the key with fingerprint BDC9 89B0 1BD2 A463 6010
>> >>A1CA 8F15 5E09 610D 69FB [3],
>> >> * all artifacts to be deployed to the Maven Central Repository [4],
>> >> * source code tag "v2.4.0-RC2" [5],
>> >> * website pull request listing the release and publishing the API
>> >> reference
>> >> manual [6].
>> >> * Java artifacts were built with Maven 3.2.5 and OpenJDK 1.8.0_112.
>> >> * Python artifact are deployed along with the source release to the
>> >> dist.apache.org [2]. If I am able to figure out how to build the
>> wheels, I
>> >> will post them there as well.
>> >>
>> >> The vote will be open for at least 72 hours. It is adopted by majority
>> >> approval, with at least 3 PMC affirmative votes.
>> >>
>> >> Thanks,
>> >> - Robert
>> >>
>> >> [1]
>> >>
>> >>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12342682=12319527
>> >> [2] https://dist.apache.org/repos/dist/dev/beam/2.4.0/
>> >> [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
>> >> [4]
>> https://repository.apache.org/content/repositories/orgapachebeam-1030/
>> >> [5] https://github.com/apache/beam/tree/v2.4.0-RC2
>> >> [6] https://github.com/apache/beam-site/pull/398
>> >
>> >
>>
>

Re: [VOTE] Release 2.4.0, release candidate #2

2018-03-08 Thread Robert Bradshaw

I put the validation checklist spreadsheet is up at
https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit?ts=5a1c7310#gid=1663314475

Regarding the direct runner regression on query 10, this is understandable
given how mutation detection has been changed for serializable coders (and
should be tracked, probably fixed by avoiding SerializableCoder). It should
not affect other runners. Could you file a bug?

Regarding waitUntilFinish, this is a bug but not a blocker--it's been this
way since teardown was introduced. There are many nice-to-haves that one
could merge from master to the release branch, but we've seen where that
trend leads.

Regarding the backwards incompatible changes in restriction tracker, this
is (as I understand it) a change to the experimental SDF API. Eugene, do
you want to comment on this?



On Thu, Mar 8, 2018 at 2:07 AM Ismaël Mejía  wrote:

> I confirm that the new release fixes both problems reported previously:
>
> - python package name
> - nexmark query 10 mutability issue with the direct runner.
>
> One extra regression is that the the fix produced a way longer
> execution time on the query.
> Not sure if a blocker but worth tracking.
>
> Query 10 - Batch/Bounded
> Version  Runtime(sec)   Events(/sec)Results
>   2.3.0   3.627609.1  1
>   2.4.0  30.8 3244.3  1
>
> Query 10 - Streaming/Unbounded
> Version  Runtime(sec)   Events(/sec)Results
>   2.3.0   6.315873.0  1
>   2.4.0 101.1  989.4  1
>
> On Thu, Mar 8, 2018 at 8:54 AM, Romain Manni-Bucau
>  wrote:
> > -1:
> > a) still consider waitUntilFinish broken and a big blocker
> > b) restrictiontracker api changed and is not backward compatible
> > (
> https://github.com/apache/beam/commit/e0034314ad196d2274cef9831ed63e090bf4d4c1#diff-098d7247eb1e9d9423bfa2ae2da38a9d
> )
> >
> > with workarounds and fixes for these two issues the other parts work
> (spark,
> > flink, direct runner, java core) on my projects
> >
> >
> >
> > Romain Manni-Bucau
> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >
> > 2018-03-08 6:26 GMT+01:00 Robert Bradshaw :
> >>
> >> Hi everyone,
> >>
> >> Please review and vote on the release candidate #2 for the version
> 2.4.0,
> >> as follows:
> >> [ ] +1, Approve the release
> >> [ ] -1, Do not approve the release (please provide specific comments)
> >>
> >> The complete staging area is available for your review, which includes:
> >> * JIRA release notes [1],
> >> * the official Apache source release to be deployed to dist.apache.org
> >> [2],
> >> which is signed with the key with fingerprint BDC9 89B0 1BD2 A463 6010
> >>A1CA 8F15 5E09 610D 69FB [3],
> >> * all artifacts to be deployed to the Maven Central Repository [4],
> >> * source code tag "v2.4.0-RC2" [5],
> >> * website pull request listing the release and publishing the API
> >> reference
> >> manual [6].
> >> * Java artifacts were built with Maven 3.2.5 and OpenJDK 1.8.0_112.
> >> * Python artifact are deployed along with the source release to the
> >> dist.apache.org [2]. If I am able to figure out how to build the
> wheels, I
> >> will post them there as well.
> >>
> >> The vote will be open for at least 72 hours. It is adopted by majority
> >> approval, with at least 3 PMC affirmative votes.
> >>
> >> Thanks,
> >> - Robert
> >>
> >> [1]
> >>
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12342682=12319527
> >> [2] https://dist.apache.org/repos/dist/dev/beam/2.4.0/
> >> [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
> >> [4]
> https://repository.apache.org/content/repositories/orgapachebeam-1030/
> >> [5] https://github.com/apache/beam/tree/v2.4.0-RC2
> >> [6] https://github.com/apache/beam-site/pull/398
> >
> >
>

Re: [VOTE] Release 2.4.0, release candidate #2

2018-03-08 Thread Ismaël Mejía

I confirm that the new release fixes both problems reported previously:

- python package name
- nexmark query 10 mutability issue with the direct runner.

One extra regression is that the the fix produced a way longer
execution time on the query.
Not sure if a blocker but worth tracking.

Query 10 - Batch/Bounded
Version  Runtime(sec)   Events(/sec)Results
  2.3.0   3.627609.1  1
  2.4.0  30.8 3244.3  1

Query 10 - Streaming/Unbounded
Version  Runtime(sec)   Events(/sec)Results
  2.3.0   6.315873.0  1
  2.4.0 101.1  989.4  1

On Thu, Mar 8, 2018 at 8:54 AM, Romain Manni-Bucau
 wrote:
> -1:
> a) still consider waitUntilFinish broken and a big blocker
> b) restrictiontracker api changed and is not backward compatible
> (https://github.com/apache/beam/commit/e0034314ad196d2274cef9831ed63e090bf4d4c1#diff-098d7247eb1e9d9423bfa2ae2da38a9d)
>
> with workarounds and fixes for these two issues the other parts work (spark,
> flink, direct runner, java core) on my projects
>
>
>
> Romain Manni-Bucau
> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>
> 2018-03-08 6:26 GMT+01:00 Robert Bradshaw :
>>
>> Hi everyone,
>>
>> Please review and vote on the release candidate #2 for the version 2.4.0,
>> as follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>>
>> The complete staging area is available for your review, which includes:
>> * JIRA release notes [1],
>> * the official Apache source release to be deployed to dist.apache.org
>> [2],
>> which is signed with the key with fingerprint BDC9 89B0 1BD2 A463 6010
>>A1CA 8F15 5E09 610D 69FB [3],
>> * all artifacts to be deployed to the Maven Central Repository [4],
>> * source code tag "v2.4.0-RC2" [5],
>> * website pull request listing the release and publishing the API
>> reference
>> manual [6].
>> * Java artifacts were built with Maven 3.2.5 and OpenJDK 1.8.0_112.
>> * Python artifact are deployed along with the source release to the
>> dist.apache.org [2]. If I am able to figure out how to build the wheels, I
>> will post them there as well.
>>
>> The vote will be open for at least 72 hours. It is adopted by majority
>> approval, with at least 3 PMC affirmative votes.
>>
>> Thanks,
>> - Robert
>>
>> [1]
>>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12342682=12319527
>> [2] https://dist.apache.org/repos/dist/dev/beam/2.4.0/
>> [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
>> [4] https://repository.apache.org/content/repositories/orgapachebeam-1030/
>> [5] https://github.com/apache/beam/tree/v2.4.0-RC2
>> [6] https://github.com/apache/beam-site/pull/398
>
>

Re: [VOTE] Release 2.4.0, release candidate #2

2018-03-07 Thread Romain Manni-Bucau

-1:
a) still consider waitUntilFinish broken and a big blocker
b) restrictiontracker api changed and is not backward compatible (
https://github.com/apache/beam/commit/e0034314ad196d2274cef9831ed63e090bf4d4c1#diff-098d7247eb1e9d9423bfa2ae2da38a9d
)

with workarounds and fixes for these two issues the other parts work
(spark, flink, direct runner, java core) on my projects



Romain Manni-Bucau
@rmannibucau  |  Blog
 | Old Blog
 | Github  |
LinkedIn  | Book


2018-03-08 6:26 GMT+01:00 Robert Bradshaw :

> Hi everyone,
>
> Please review and vote on the release candidate #2 for the version 2.4.0,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org
> [2],
> which is signed with the key with fingerprint BDC9 89B0 1BD2 A463 6010
>A1CA 8F15 5E09 610D 69FB [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.4.0-RC2" [5],
> * website pull request listing the release and publishing the API reference
> manual [6].
> * Java artifacts were built with Maven 3.2.5 and OpenJDK 1.8.0_112.
> * Python artifact are deployed along with the source release to the
> dist.apache.org [2]. If I am able to figure out how to build the wheels, I
> will post them there as well.
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> - Robert
>
> [1]
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> version=12342682=12319527
> [2] https://dist.apache.org/repos/dist/dev/beam/2.4.0/
> [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1030/
> [5] https://github.com/apache/beam/tree/v2.4.0-RC2
> [6] https://github.com/apache/beam-site/pull/398
>

[VOTE] Release 2.4.0, release candidate #2

2018-03-07 Thread Robert Bradshaw

Hi everyone,

Please review and vote on the release candidate #2 for the version 2.4.0,
as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release to be deployed to dist.apache.org [2],
which is signed with the key with fingerprint BDC9 89B0 1BD2 A463 6010
   A1CA 8F15 5E09 610D 69FB [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "v2.4.0-RC2" [5],
* website pull request listing the release and publishing the API reference
manual [6].
* Java artifacts were built with Maven 3.2.5 and OpenJDK 1.8.0_112.
* Python artifact are deployed along with the source release to the
dist.apache.org [2]. If I am able to figure out how to build the wheels, I
will post them there as well.

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
- Robert

[1]
https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12342682=12319527
[2] https://dist.apache.org/repos/dist/dev/beam/2.4.0/
[3] https://dist.apache.org/repos/dist/dev/beam/KEYS
[4] https://repository.apache.org/content/repositories/orgapachebeam-1030/
[5] https://github.com/apache/beam/tree/v2.4.0-RC2
[6] https://github.com/apache/beam-site/pull/398

Re: [VOTE] Release 2.4.0, release candidate #2

Re: [VOTE] Release 2.4.0, release candidate #2

Re: [VOTE] Release 2.4.0, release candidate #2

Re: [VOTE] Release 2.4.0, release candidate #2

Re: [VOTE] Release 2.4.0, release candidate #2

Re: [VOTE] Release 2.4.0, release candidate #2

Re: [VOTE] Release 2.4.0, release candidate #2

Re: [VOTE] Release 2.4.0, release candidate #2

Re: [VOTE] Release 2.4.0, release candidate #2

Re: [VOTE] Release 2.4.0, release candidate #2

Re: [VOTE] Release 2.4.0, release candidate #2

Re: [VOTE] Release 2.4.0, release candidate #2

Re: [VOTE] Release 2.4.0, release candidate #2

Re: [VOTE] Release 2.4.0, release candidate #2

Re: [VOTE] Release 2.4.0, release candidate #2

Re: [VOTE] Release 2.4.0, release candidate #2

Re: [VOTE] Release 2.4.0, release candidate #2

Re: [VOTE] Release 2.4.0, release candidate #2

Re: [VOTE] Release 2.4.0, release candidate #2

[VOTE] Release 2.4.0, release candidate #2

20 matches

Site Navigation

Mail list logo

Footer information