Re: [VOTE] Release 2.4.0, release candidate #2

Just to give an update on this, I plan on creating an RC3 as soon as I get the dataflow containers rebuilt. (Been busy with the beam summit among other things.) Except for direct runner fixes and the dataflow tags, I expect it to be the same as RC2, so testing on flink/apex/spark done now would

Re: [VOTE] Release 2.4.0, release candidate #2

Thank you Valentyn for reporting this. I have traced the issue back to https://github.com/apache/beam/pull/4666, so I have sent out a PR to fix: https://github.com/apache/beam/pull/4846. On Fri, Mar 9, 2018 at 2:17 PM, Valentyn Tymofieiev wrote: > -1. > > Checked Python

Re: [VOTE] Release 2.4.0, release candidate #2

-1. Checked Python Quickstarts (Passed) and Python MobileGaming on DirectRunner. I observe an issue in BQ sink for hourly teams score example: https://issues.apache.org/jira/browse/BEAM-3824 On Fri, Mar 9, 2018 at 10:49 AM, Lukasz Cwik wrote: > I checked that word count

Re: [VOTE] Release 2.4.0, release candidate #2

I checked that word count quickstarts (except Dataflow) worked for RC2 to hopefully prevent an RC4. On Fri, Mar 9, 2018 at 10:29 AM, Robert Bradshaw wrote: > Thanks, Alan, for pointing this out. I see this now, and it looks like I > need to finish building the dataflow

Re: [VOTE] Release 2.4.0, release candidate #2

Thanks, Alan, for pointing this out. I see this now, and it looks like I need to finish building the dataflow workers so they have something to point to. I will do this and release an RC3 once that's ready. In the meantime, it'd be great if we could validate everything else about this RC such

Re: [VOTE] Release 2.4.0, release candidate #2

Also, I figured out how to create Python wheels and put them up at https://test.pypi.org/project/apache-beam/#files . I'm in the process of documenting and cleaning up this process. On Wed, Mar 7, 2018 at 9:26 PM Robert Bradshaw wrote: > Hi everyone, > > Please review and

Re: [VOTE] Release 2.4.0, release candidate #2

The symptoms were a Coder that did not properly implement structuralValue (e.g. it returned the original object whose equals() was not overridden) but that did have deterministic serialization. Here at every element we're comparing serialized forms to ensure the elements were not mutated, which is

Re: [VOTE] Release 2.4.0, release candidate #2

This makes me worry that the performance regression is elsewhere, in which case we should investigate. On Fri, Mar 9, 2018 at 1:02 AM Etienne Chauchot wrote: > Le jeudi 08 mars 2018 à 20:01 +, Reuven Lax a écrit : > > Does Nexmark use SerializableCoder? > > > Actually

Re: [VOTE] Release 2.4.0, release candidate #2

Le jeudi 08 mars 2018 à 20:01 +, Reuven Lax a écrit : > Does Nexmark use SerializableCoder? Actually SerializableCoder is registered in Nexmark but the default and the current configuration are set to use "by hand serialization". See: public static void setupPipeline(CoderStrategy

Re: [VOTE] Release 2.4.0, release candidate #2

Yes, the release guide has a segment "Update release specific configurations" that has a tidbit about this. On Thu, Mar 8, 2018 at 3:45 PM, Alan Myrvold wrote: > The dataflow java worker version wasn't updated on the branch as in past > releases ... should it be? >

Re: [VOTE] Release 2.4.0, release candidate #2

The dataflow java worker version wasn't updated on the branch as in past releases ... should it be? https://issues.apache.org/jira/browse/BEAM-3815 On Thu, Mar 8, 2018 at 1:40 PM Romain Manni-Bucau wrote: > Can still be provided as a generic one (like the an offset or

Re: [VOTE] Release 2.4.0, release candidate #2

Can still be provided as a generic one (like the an offset or key based one) but good enough for now, right, was just surprising to not see it when checking the breakage. Le 8 mars 2018 22:05, "Eugene Kirpichov" a écrit : All SDF-related method annotations in DoFn are

Re: [VOTE] Release 2.4.0, release candidate #2

All SDF-related method annotations in DoFn are marked @Experimental. I guess that should apply to RestrictionTracker too, but I wouldn't be too worried about that, since it only makes sense to use in the context of those methods. On Thu, Mar 8, 2018 at 12:36 PM Romain Manni-Bucau

Re: [VOTE] Release 2.4.0, release candidate #2

Hmm, does sdf api misses some @Experimental then? To clarify: for waitUntilFinish I'm fine with the 2.4 as this but cant +1 or +0 since none of my tests pass reliably in current state without a retry strategy making the call useless. Le 8 mars 2018 21:02, "Reuven Lax" a écrit

Re: [VOTE] Release 2.4.0, release candidate #2

On Thu, Mar 8, 2018 at 12:02 PM Reuven Lax wrote: > Does Nexmark use SerializableCoder? > That's what the errors (and fix) for RC1 seemed to indicate. > On Thu, Mar 8, 2018 at 10:42 AM Robert Bradshaw > wrote: > >> I put the validation checklist

Re: [VOTE] Release 2.4.0, release candidate #2

Yes, SDF is an experimental API at this point, so backwards incompatible changes are allowed and expected. On Thu, Mar 8, 2018, 10:42 AM Robert Bradshaw wrote: > I put the validation checklist spreadsheet is up at >

Re: [VOTE] Release 2.4.0, release candidate #2

I put the validation checklist spreadsheet is up at https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit?ts=5a1c7310#gid=1663314475 Regarding the direct runner regression on query 10, this is understandable given how mutation detection has been changed for

Re: [VOTE] Release 2.4.0, release candidate #2

I confirm that the new release fixes both problems reported previously: - python package name - nexmark query 10 mutability issue with the direct runner. One extra regression is that the the fix produced a way longer execution time on the query. Not sure if a blocker but worth tracking. Query

Re: [VOTE] Release 2.4.0, release candidate #2

-1: a) still consider waitUntilFinish broken and a big blocker b) restrictiontracker api changed and is not backward compatible ( https://github.com/apache/beam/commit/e0034314ad196d2274cef9831ed63e090bf4d4c1#diff-098d7247eb1e9d9423bfa2ae2da38a9d ) with workarounds and fixes for these two issues

[VOTE] Release 2.4.0, release candidate #2

Hi everyone, Please review and vote on the release candidate #2 for the version 2.4.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], *