Cherrypick created: https://github.com/apache/beam/pull/5607 Tests still running.
On Tue, Jun 12, 2018 at 9:59 AM Udi Meiri <eh...@google.com> wrote: > -1: Would like to cherry pick a fix for > https://issues.apache.org/jira/browse/BEAM-4536 > > On Tue, Jun 12, 2018 at 9:48 AM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > >> Any update about your vote and fix ? >> >> Thanks ! >> Regards >> JB >> >> On 12/06/2018 04:02, Udi Meiri wrote: >> > Another bug: reading from PubSub with_attributes=True is broken on >> > Python with Dataflow. >> > https://issues.apache.org/jira/browse/BEAM-4536 >> > >> > JB, I'm making a PR that removes this keyword and I'd like to propose it >> > as a cherrypick to 2.5.0. >> > (feature should be fixed in the next release) >> > >> > On Mon, Jun 11, 2018 at 6:19 PM Chamikara Jayalath < >> chamik...@google.com >> > <mailto:chamik...@google.com>> wrote: >> > >> > FYI: looks like Python tests are failing for Windows. JIRA >> > is https://issues.apache.org/jira/browse/BEAM-4535. >> > >> > I don't think this is a release blocker but this should probably go >> > in release notes (for any user that tries to run tests on Python >> > source build). And we should try to incorporate a fix if we happen >> > to cut another release candidate for some reason. >> > >> > Thanks, >> > Cham >> > >> > On Mon, Jun 11, 2018 at 5:46 PM Pablo Estrada <pabl...@google.com >> > <mailto:pabl...@google.com>> wrote: >> > >> > Thanks everyone who has pitched in to validate the release! >> > >> > Boyuan Zhang and I have also run a few pipelines, and verified >> > that they work properly (see release validation spreadsheet[1]). >> > >> > We have also found that the Game Stats pipeline is failing in >> > Python Streaming Dataflow. I have filed BEAM-4534[2]. This is >> > not a blocker, since Python streaming is not yet fully >> supported. >> > >> > It seems that the uploaded artifacts look good. >> > >> > We have noticed that the Python artifacts are still missing >> > Python wheel files (compare [3] and [4]). JB, could you please >> > add the wheel files? Boyuan and I can try to help you prepare >> > them / upload them if necessary. Please let us know. >> > >> > Thanks again! >> > -P. >> > >> > [1] >> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=152451807 >> > [2] https://issues.apache.org/jira/browse/BEAM-4534 >> > [3] https://dist.apache.org/repos/dist/dev/beam/2.4.0/ >> > [4] https://dist.apache.org/repos/dist/dev/beam/2.5.0/ >> > >> > On Mon, Jun 11, 2018 at 12:37 PM Alan Myrvold >> > <amyrv...@google.com <mailto:amyrv...@google.com>> wrote: >> > >> > +1 (non-binding) >> > >> > tested some of the quickstarts >> > >> > On Sun, Jun 10, 2018 at 1:39 AM Tim >> > <timrobertson...@gmail.com >> > <mailto:timrobertson...@gmail.com>> wrote: >> > >> > Tested by our team: >> > - mvn inclusion >> > - Avro, ES, Hadoop IF IO >> > - Pipelines run on Spark (Cloudera 5.12.0 YARN cluster) >> > - Reviewed release notes >> > >> > +1 >> > >> > Thanks also to everyone who helped get over the gradle >> > hurdle and in particular to JB. >> > >> > Tim >> > >> > > On 9 Jun 2018, at 05:56, Jean-Baptiste Onofré >> > <j...@nanthrax.net <mailto:j...@nanthrax.net>> wrote: >> > > >> > > No problem Pablo. >> > > >> > > The vote period is a minimum, it can be extended as >> > requested or if we >> > > don't have the minimum of 3 binding votes. >> > > >> > > Regards >> > > JB >> > > >> > >> On 09/06/2018 01:54, Pablo Estrada wrote: >> > >> Hello all, >> > >> I'd like to request an extension of the voting period >> > until Monday >> > >> evening (US time, so later in other geographical >> > regions). This is >> > >> because we were only now able to publish Dataflow >> > Workers, and have not >> > >> had the chance to run release validation tests on >> > them. The extension >> > >> will allow us to validate and vote by Monday. >> > >> Is this acceptable to the community? >> > >> >> > >> Best >> > >> -P. >> > >> >> > >> On Fri, Jun 8, 2018 at 6:20 AM Alexey Romanenko >> > >> <aromanenko....@gmail.com >> > <mailto:aromanenko....@gmail.com> >> > <mailto:aromanenko....@gmail.com >> > <mailto:aromanenko....@gmail.com>>> wrote: >> > >> >> > >> Thank you JB for your work! >> > >> >> > >> I tested running simple streaming (/KafkaIO/) and >> > batch (/TextIO / >> > >> HDFS/) pipelines with SparkRunner on YARN cluster >> > - it works fine. >> > >> >> > >> WBR, >> > >> Alexey >> > >> >> > >> >> > >>> On 8 Jun 2018, at 10:00, Etienne Chauchot >> > <echauc...@apache.org <mailto:echauc...@apache.org> >> > >>> <mailto:echauc...@apache.org >> > <mailto:echauc...@apache.org>>> wrote: >> > >>> >> > >>> I forgot to vote: >> > >>> +1 (non binding). >> > >>> What I tested: >> > >>> - no functional or performance regression >> > comparing to v2.4 >> > >>> - dependencies in the poms are ok >> > >>> >> > >>> Etienne >> > >>>> Le vendredi 08 juin 2018 à 08:27 +0200, Romain >> > Manni-Bucau a écrit : >> > >>>> +1 (non-binding), mainstream usage is not broken >> > by the pom >> > >>>> changes and runtime has no known regression >> > compared to the 2.4.0 >> > >>>> >> > >>>> (side note: kudo to JB for this build tool >> > change release, I know >> > >>>> how it can hurt ;)) >> > >>>> >> > >>>> Romain Manni-Bucau >> > >>>> @rmannibucau <https://twitter.com/rmannibucau> >> > | Blog >> > >>>> <https://rmannibucau.metawerx.net/> | Old Blog >> > >>>> <http://rmannibucau.wordpress.com/> | Github >> > >>>> <https://github.com/rmannibucau> | LinkedIn >> > >>>> <https://www.linkedin.com/in/rmannibucau> | >> Book >> > >>>> >> > < >> https://www.packtpub.com/application-development/java-ee-8-high-performance >> > >> > >>>> >> > >>>> >> > >>>> Le jeu. 7 juin 2018 à 16:17, Jean-Baptiste >> Onofré >> > >>>> <j...@nanthrax.net <mailto:j...@nanthrax.net> >> > <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net>>> a >> écrit : >> > >>>>> Thanks for the details Etienne ! >> > >>>>> >> > >>>>> The good news is that the artifacts seem OK and >> > the overall Nexmark >> > >>>>> results are consistent with the 2.4.0 release >> ones. >> > >>>>> >> > >>>>> I'm starting a complete review using the >> > beam-samples as well. >> > >>>>> >> > >>>>> Regards >> > >>>>> JB >> > >>>>> >> > >>>>>> On 07/06/2018 16:14, Etienne Chauchot wrote: >> > >>>>>> Hi, >> > >>>>>> I've just run the nexmark queries on v2.5.0-RC1 >> tag >> > >>>>>> What we can notice: >> > >>>>>> - query 3 (exercises CoGroupByKey, state and >> > timer) shows >> > >>>>> different >> > >>>>>> output with DR between batch and streaming and >> > with the other >> > >>>>> runners => >> > >>>>>> I compared with v2.4 there were still these >> > differences but with >> > >>>>>> different output size numbers >> > >>>>>> >> > >>>>>> - query 6 (exercises specialized combiner) shows >> > different output >> > >>>>>> between the runners => the correct output is 401. >> > strange that >> > >>>>> in batch >> > >>>>>> mode some runners output les Sellers. I compared >> > with v2.4 >> > >>>>> same output >> > >>>>>> >> > >>>>>> - response time of query 7 (exercices Max >> > transform, fanout >> > >>>>> and side >> > >>>>>> input) is very slow on DR => I compared with v2.4 >> > , comparable >> > >>>>> execution >> > >>>>>> times >> > >>>>>> >> > >>>>>> I'm not comparing q10 because it is a write to >> > GCS so it is >> > >>>>> very specific. >> > >>>>>> >> > >>>>>> => Basically no regression comparing to v2.4 >> > >>>>>> >> > >>>>>> For the record here is the output (waiting for >> > ongoing perfkit >> > >>>>> integration): >> > >>>>>> >> > >>>>>> >> > >>>>>> 1. DR batch >> > >>>>>> >> > >>>>>> Performance: >> > >>>>>> >> > >>>>>> >> > >>>>> Conf Runtime(sec) (Baseline) Events(/sec) >> > (Baseline) Results (Baseline) >> > >>>>>> >> > >>>>>> >> > >>>>> 0000 5,8 17283,1 >> > 100000 >> > >>>>>> >> > >>>>>> >> > >>>>> 0001 3,2 31104,2 >> > 92000 >> > >>>>>> >> > >>>>>> >> > >>>>> 0002 1,2 82918,7 >> > 351 >> > >>>>>> >> > >>>>>> >> > >>>>> 0003 2,2 46210,7 >> > 458 >> > >>>>>> >> > >>>>>> >> > >>>>> 0004 1,2 8503,4 >> > 40 >> > >>>>>> >> > >>>>>> >> > >>>>> 0005 4,0 25220,7 >> > 12 >> > >>>>>> >> > >>>>>> >> > >>>>> 0006 0,9 11148,3 >> > 401 >> > >>>>>> >> > >>>>>> >> > >>>>> 0007 13,2 7580,9 >> > 1 >> > >>>>>> >> > >>>>>> >> > >>>>> 0008 1,5 67340,1 >> > 6000 >> > >>>>>> >> > >>>>>> >> > >>>>> 0009 0,7 14025,2 >> > 298 >> > >>>>>> >> > >>>>>> >> > >>>>> 0010 12,8 7793,0 >> > 1 >> > >>>>>> >> > >>>>>> >> > >>>>> 0011 2,4 42319,1 >> > 1919 >> > >>>>>> >> > >>>>>> >> > >>>>> 0012 1,6 61462,8 >> > 1919 >> > >>>>>> >> > >>>>> >> > >> >> ========================================================================================== >> > >>>>>> >> > >>>>>> 2. DR streaming >> > >>>>>> >> > >>>>>> Performance: >> > >>>>>> >> > >>>>>> >> > >>>>> Conf Runtime(sec) (Baseline) Events(/sec) >> > (Baseline) Results (Baseline) >> > >>>>>> >> > >>>>>> >> > >>>>> 0000 6,5 15285,8 >> > 100000 >> > >>>>>> >> > >>>>>> >> > >>>>> 0001 3,7 27397,3 >> > 92000 >> > >>>>>> >> > >>>>>> >> > >>>>> 0002 1,4 69108,5 >> > 351 >> > >>>>>> >> > >>>>>> >> > >>>>> 0003 3,2 31181,8 >> > 447 >> > >>>>>> >> > >>>>>> >> > >>>>> 0004 1,2 8361,2 >> > 40 >> > >>>>>> >> > >>>>>> >> > >>>>> 0005 5,3 18903,6 >> > 12 >> > >>>>>> >> > >>>>>> >> > >>>>> 0006 0,9 11111,1 >> > 401 >> > >>>>>> >> > >>>>>> >> > >>>>> 0007 82,5 1212,2 >> > 1 >> > >>>>>> >> > >>>>>> >> > >>>>> 0008 2,0 51072,5 >> > 6000 >> > >>>>>> >> > >>>>>> >> > >>>>> 0009 0,8 12903,2 >> > 298 >> > >>>>>> >> > >>>>>> >> > >>>>> 0010 49,5 2021,8 >> > 1 >> > >>>>>> >> > >>>>>> >> > >>>>> 0011 3,9 25667,4 >> > 1919 >> > >>>>>> >> > >>>>>> >> > >>>>> 0012 2,4 41067,8 >> > 1919 >> > >>>>>> >> > >>>>> >> > >> >> ========================================================================================== >> > >>>>>> >> > >>>>>> 3. Flink batch >> > >>>>>> Performance: >> > >>>>>> >> > >>>>>> >> > >>>>> Conf Runtime(sec) (Baseline) Events(/sec) >> > (Baseline) Results (Baseline) >> > >>>>>> >> > >>>>>> >> > >>>>> 0000 1,0 97656,3 >> > 100000 >> > >>>>>> >> > >>>>>> >> > >>>>> 0001 0,7 141643,1 >> > 92000 >> > >>>>>> >> > >>>>>> >> > >>>>> 0002 0,4 228310,5 >> > 351 >> > >>>>>> >> > >>>>>> >> > >>>>> 0003 1,6 64020,5 >> > 580 >> > >>>>>> >> > >>>>>> >> > >>>>> 0004 0,7 13831,3 >> > 40 >> > >>>>>> >> > >>>>>> >> > >>>>> 0005 1,4 72939,5 >> > 12 >> > >>>>>> >> > >>>>>> >> > >>>>> 0006 0,5 20491,8 >> > 103 >> > >>>>>> >> > >>>>>> >> > >>>>> 0007 1,3 74239,0 >> > 1 >> > >>>>>> >> > >>>>>> >> > >>>>> 0008 0,8 121506,7 >> > 6000 >> > >>>>>> >> > >>>>>> >> > >>>>> 0009 0,6 17953,3 >> > 298 >> > >>>>>> >> > >>>>>> >> > >>>>> 0010 1,3 74682,6 >> > 1 >> > >>>>>> >> > >>>>>> >> > >>>>> 0011 1,1 92936,8 >> > 1919 >> > >>>>>> >> > >>>>>> >> > >>>>> 0012 0,8 123001,2 >> > 1919 >> > >>>>>> >> > >>>>> >> > >> >> ========================================================================================== >> > >>>>>> >> > >>>>>> 4. Flink streaming >> > >>>>>> Performance: >> > >>>>>> >> > >>>>>> >> > >>>>> Conf Runtime(sec) (Baseline) Events(/sec) >> > (Baseline) Results (Baseline) >> > >>>>>> >> > >>>>>> >> > >>>>> 0000 5,4 18677,6 >> > 100000 >> > >>>>>> >> > >>>>>> >> > >>>>> 0001 2,8 35511,4 >> > 92000 >> > >>>>>> >> > >>>>>> >> > >>>>> 0002 1,8 54318,3 >> > 351 >> > >>>>>> >> > >>>>>> >> > >>>>> 0003 2,4 41614,6 >> > 580 >> > >>>>>> >> > >>>>>> >> > >>>>> 0004 1,0 10341,3 >> > 40 >> > >>>>>> >> > >>>>>> >> > >>>>> 0005 3,4 29568,3 >> > 12 >> > >>>>>> >> > >>>>>> >> > >>>>> 0006 0,7 13369,0 >> > 401 >> > >>>>>> >> > >>>>>> >> > >>>>> 0007 2,8 36192,5 >> > 1 >> > >>>>>> >> > >>>>>> >> > >>>>> 0008 1,8 54854,6 >> > 6000 >> > >>>>>> >> > >>>>>> >> > >>>>> 0009 0,7 13369,0 >> > 298 >> > >>>>>> >> > >>>>>> >> > >>>>> 0010 3,4 29841,8 >> > 2 >> > >>>>>> >> > >>>>>> >> > >>>>> 0011 5,0 19932,2 >> > 1919 >> > >>>>>> >> > >>>>>> >> > >>>>> 0012 2,6 38835,0 >> > 1919 >> > >>>>>> >> > >>>>> >> > >> >> ========================================================================================== >> > >>>>>> >> > >>>>>> 5. Spark batch >> > >>>>>> Performance: >> > >>>>>> >> > >>>>>> >> > >>>>> Conf Runtime(sec) (Baseline) Events(/sec) >> > (Baseline) Results (Baseline) >> > >>>>>> >> > >>>>>> >> > >>>>> 0000 1,5 65445,0 >> > 100000 >> > >>>>>> >> > >>>>>> >> > >>>>> 0001 1,3 79491,3 >> > 92000 >> > >>>>>> >> > >>>>>> >> > >>>>> 0002 0,9 112107,6 >> > 351 >> > >>>>>> >> > >>>>>> >> > >>>>> 0003 2,0 48804,3 >> > 580 >> > >>>>>> >> > >>>>>> >> > >>>>> 0004 1,2 8382,2 >> > 40 >> > >>>>>> >> > >>>>>> >> > >>>>> 0005 2,0 50838,8 >> > 12 >> > >>>>>> >> > >>>>>> >> > >>>>> 0006 1,0 9699,3 >> > 103 >> > >>>>>> >> > >>>>>> >> > >>>>> 0007 2,3 43308,8 >> > 1 >> > >>>>>> >> > >>>>>> >> > >>>>> 0008 2,1 46794,6 >> > 6000 >> > >>>>>> >> > >>>>>> >> > >>>>> 0009 1,1 8976,7 >> > 298 >> > >>>>>> >> > >>>>>> >> > >>>>> 0010 1,6 62111,8 >> > 1 >> > >>>>>> >> > >>>>>> >> > >>>>> 0011 2,1 46598,3 >> > 1919 >> > >>>>>> >> > >>>>>> >> > >>>>> 0012 2,3 43687,2 >> > 1919 >> > >>>>>> >> > >>>>> >> > >> >> ========================================================================================== >> > >>>>>> >> > >>>>>>> Le mercredi 06 juin 2018 à 10:50 +0200, Etienne >> > Chauchot a écrit : >> > >>>>>>> Thanks JB for all your work ! I believe doing >> > the first >> > >>>>> gradle release >> > >>>>>>> must have been hard. >> > >>>>>>> I'll run Nexmark on the release and keep you >> posted. >> > >>>>>>> >> > >>>>>>> Best >> > >>>>>>> Etienne >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> Le mercredi 06 juin 2018 à 10:44 +0200, >> > Jean-Baptiste Onofré >> > >>>>> a écrit : >> > >>>>>>>> Hi everyone, >> > >>>>>>>> >> > >>>>>>>> Please review and vote on the release candidate >> > #1 for the >> > >>>>> version >> > >>>>>>>> 2.5.0, as follows: >> > >>>>>>>> >> > >>>>>>>> [ ] +1, Approve the release >> > >>>>>>>> [ ] -1, Do not approve the release (please >> > provide specific >> > >>>>> comments) >> > >>>>>>>> >> > >>>>>>>> NB: this is the first release using Gradle, so >> > don't be too >> > >>>>> harsh ;) A >> > >>>>>>>> PR about the release guide will follow thanks >> > to this release. >> > >>>>>>>> >> > >>>>>>>> The complete staging area is available for your >> > review, >> > >>>>> which includes: >> > >>>>>>>> * JIRA release notes [1], >> > >>>>>>>> * the official Apache source release to be >> > deployed to >> > >>>>> dist.apache.org <http://dist.apache.org> >> > <http://dist.apache.org/> >> > >>>>>>>> [2], which is signed with the key with >> > fingerprint C8282E76 [3], >> > >>>>>>>> * all artifacts to be deployed to the Maven >> Central >> > >>>>> Repository [4], >> > >>>>>>>> * source code tag "v2.5.0-RC1" [5], >> > >>>>>>>> * website pull request listing the release and >> > publishing >> > >>>>> the API >> > >>>>>>>> reference manual [6]. >> > >>>>>>>> * Java artifacts were built with Gradle 4.7 >> > (wrapper) and >> > >>>>> OpenJDK/Oracle >> > >>>>>>>> JDK 1.8.0_172 (Oracle Corporation 25.172-b11). >> > >>>>>>>> * Python artifacts are deployed along with the >> > source >> > >>>>> release to the >> > >>>>>>>> dist.apache.org <http://dist.apache.org> >> > <http://dist.apache.org/> [2]. >> > >>>>>>>> >> > >>>>>>>> The vote will be open for at least 72 hours. It >> > is adopted >> > >>>>> by majority >> > >>>>>>>> approval, with at least 3 PMC affirmative >> votes. >> > >>>>>>>> >> > >>>>>>>> Thanks, >> > >>>>>>>> JB >> > >>>>>>>> >> > >>>>>>>> [1] >> > >>>>>>>> >> > >>>>> >> > >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847 >> > >>>>>>>> [2] >> > https://dist.apache.org/repos/dist/dev/beam/2.5.0/ >> > >>>>>>>> [3] >> > https://dist.apache.org/repos/dist/release/beam/KEYS >> > >>>>>>>> [4] >> > >>>>> >> > >> https://repository.apache.org/content/repositories/orgapachebeam-1041/ >> > >>>>>>>> [5] >> https://github.com/apache/beam/tree/v2.5.0-RC1 >> > >>>>>>>> [6] >> https://github.com/apache/beam-site/pull/463 >> > >>>>>>>> >> > >>>>> >> > >> >> > >> -- >> > >> Got feedback? go/pabloem-feedback >> <https://goto.google.com/pabloem-feedback> >> > <https://goto.google.com/pabloem-feedback> >> > > >> > > -- >> > > Jean-Baptiste Onofré >> > > jbono...@apache.org <mailto:jbono...@apache.org> >> > > http://blog.nanthrax.net >> > > Talend - http://www.talend.com >> > >> > -- >> > Got feedback? go/pabloem-feedback >> <https://goto.google.com/pabloem-feedback> >> > <https://goto.google.com/pabloem-feedback> >> > >> >> -- >> Jean-Baptiste Onofré >> jbono...@apache.org >> http://blog.nanthrax.net >> Talend - http://www.talend.com >> >
smime.p7s
Description: S/MIME Cryptographic Signature
