Changes are cherrypicked to the release-2.3.0 branch, or applied directly there.
On Tue, Feb 6, 2018 at 2:08 PM, Eugene Kirpichov <[email protected]> wrote: > Possibly stupid question: are new RCs created from master, or by > cherrypicks of desired changes on top of previous RCs? > (it affects whether or not my recently merged change on "master" will be > in 2.3.0; I'd like it to be :) ) > > On Tue, Feb 6, 2018 at 1:49 PM Lukasz Cwik <[email protected]> wrote: > >> +1 for rollback as well. >> >> On Tue, Feb 6, 2018 at 12:37 PM, Reuven Lax <[email protected]> wrote: >> >>> +1 for the rollback. >>> >>> On Tue, Feb 6, 2018 at 9:24 AM, Kenneth Knowles <[email protected]> wrote: >>> >>>> +1 for option 1. It is almost certainly repeated proto parsing. That is >>>> fixed for ParDo via generalized caching but not for other transforms. Flink >>>> also has a proto round trip, but the design might avoid the problem anyhow. >>>> We should investigate before releasing, or just preemptively roll it back >>>> too. The proto round trips are helpful for maturing portability and for >>>> preventing hacks from returning but otherwise don't matter so much yet. >>>> >>>> Kenn >>>> >>>> On Tue, Feb 6, 2018 at 8:54 AM, Jean-Baptiste Onofré <[email protected]> >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> As you might have seen, I found the cause of performance degradation >>>>> on the >>>>> direct runner (BEAM-3617). >>>>> >>>>> We have basically three options for RC2: >>>>> >>>>> 1. We revert the change for 2.3.0. I think it's the fastest and more >>>>> secure way. >>>>> I created a PR for that (https://github.com/apache/beam/pull/4609). >>>>> It gives us >>>>> time to investigate and provide a better fix in Runner API for 2.4.0. >>>>> >>>>> 2. If the fix is easy and with limited impact in Runner API, we do >>>>> that. >>>>> >>>>> 3. We leave as it is, adding a note that direct runner has worse >>>>> performance in >>>>> 2.3.0 compared to 2.2.0. >>>>> >>>>> In any case, I would like to cut RC2 later tonight or tomorrow morning >>>>> (my time). >>>>> >>>>> Thoughts ? >>>>> >>>>> Regards >>>>> JB >>>>> >>>>> On 02/06/2018 09:42 AM, Jean-Baptiste Onofré wrote: >>>>> > Hi Reuven, >>>>> > >>>>> > it's what I'm suspecting. git bisect should give us more information >>>>> (still in >>>>> > progress, 3 more steps to complete ;)). I keep you posted. >>>>> > >>>>> > Regards >>>>> > JB >>>>> > >>>>> > On 02/06/2018 08:36 AM, Reuven Lax wrote: >>>>> >> Could this be related to any of the portability changes? >>>>> >> >>>>> >> On Mon, Feb 5, 2018 at 7:51 AM, Jean-Baptiste Onofré < >>>>> [email protected] >>>>> >> <mailto:[email protected]>> wrote: >>>>> >> >>>>> >> Created: >>>>> >> >>>>> >> https://issues.apache.org/jira/browse/BEAM-3617 >>>>> >> <https://issues.apache.org/jira/browse/BEAM-3617> >>>>> >> >>>>> >> Regards >>>>> >> JB >>>>> >> >>>>> >> On 02/05/2018 04:42 PM, Kenneth Knowles wrote: >>>>> >> > What is the Jira for direct runner perf? >>>>> >> > >>>>> >> > On Mon, Feb 5, 2018 at 4:35 AM, Jean-Baptiste Onofré < >>>>> [email protected] <mailto:[email protected]> >>>>> >> > <mailto:[email protected] <mailto:[email protected]>>> wrote: >>>>> >> > >>>>> >> > Thanks ! >>>>> >> > >>>>> >> > I cherry-pick on release-2.3.0 branch. >>>>> >> > >>>>> >> > I'm on the direct runner perf test in the mean time. >>>>> >> > >>>>> >> > Thanks again ! >>>>> >> > >>>>> >> > Regards >>>>> >> > JB >>>>> >> > >>>>> >> > On 02/05/2018 12:06 PM, Aljoscha Krettek wrote: >>>>> >> > > I merged fixes for: >>>>> >> > > - https://issues.apache.org/jira/browse/BEAM-3186 >>>>> >> <https://issues.apache.org/jira/browse/BEAM-3186> >>>>> >> > <https://issues.apache.org/jira/browse/BEAM-3186 >>>>> >> <https://issues.apache.org/jira/browse/BEAM-3186>> >>>>> >> > > - https://issues.apache.org/jira/browse/BEAM-3589 >>>>> >> <https://issues.apache.org/jira/browse/BEAM-3589> >>>>> >> > <https://issues.apache.org/jira/browse/BEAM-3589 >>>>> >> <https://issues.apache.org/jira/browse/BEAM-3589>> >>>>> >> > > >>>>> >> > > @JB I didn't yet merge them on the 2.3.0 branch, >>>>> though, but I can or you can go >>>>> >> > > ahead. >>>>> >> > > >>>>> >> > >> On 5. Feb 2018, at 06:30, Jean-Baptiste Onofré < >>>>> [email protected] <mailto:[email protected]> <mailto:[email protected] >>>>> >> <mailto:[email protected]>> >>>>> >> > >> <mailto:[email protected] <mailto:[email protected]> >>>>> >> <mailto:[email protected] <mailto:[email protected]>>>> wrote: >>>>> >> > >> >>>>> >> > >> Hi guys, >>>>> >> > >> >>>>> >> > >> Quick update on the RC2 preparation: >>>>> >> > >> >>>>> >> > >> * BEAM-3587 (TextIO with Flink) seems related to a >>>>> custom build with >>>>> >> > Gradle (not >>>>> >> > >> using artifacts created by Maven). Anyway, I will take >>>>> a look today. >>>>> >> > >> * BEAM-3186 has a PR. Aljoscha will do the review >>>>> pretty soon. >>>>> >> > >> * I'm also taking a look on the direct runner >>>>> performance change. >>>>> >> > >> >>>>> >> > >> The goal is to cut RC2 tomorrow or the day after. >>>>> >> > >> >>>>> >> > >> Regards >>>>> >> > >> JB >>>>> >> > >> >>>>> >> > >> On 02/01/2018 04:16 PM, Jean-Baptiste Onofré wrote: >>>>> >> > >>> Hi guys, >>>>> >> > >>> >>>>> >> > >>> Especially due to BEAM-3587 & BEAM-3186 regressions, >>>>> I cancel RC1. >>>>> >> > >>> >>>>> >> > >>> We will cherry-pick fixes on release-2.3.0 branch. >>>>> >> > >>> >>>>> >> > >>> I'm updating Jira right now. When the fixes will be >>>>> cherry-picked, I >>>>> >> > will submit >>>>> >> > >>> a RC2 to vote. >>>>> >> > >>> >>>>> >> > >>> Thanks ! >>>>> >> > >>> Regards >>>>> >> > >>> JB >>>>> >> > >>> >>>>> >> > >>> On 01/30/2018 09:04 AM, Jean-Baptiste Onofré wrote: >>>>> >> > >>>> Hi everyone, >>>>> >> > >>>> >>>>> >> > >>>> Please review and vote on the release candidate #1 >>>>> for the version >>>>> >> > 2.3.0, as >>>>> >> > >>>> follows: >>>>> >> > >>>> >>>>> >> > >>>> [ ] +1, Approve the release >>>>> >> > >>>> [ ] -1, Do not approve the release (please provide >>>>> specific comments) >>>>> >> > >>>> >>>>> >> > >>>> >>>>> >> > >>>> The complete staging area is available for your >>>>> review, which >>>>> >> includes: >>>>> >> > >>>> * JIRA release notes [1], >>>>> >> > >>>> * the official Apache source release to be deployed >>>>> to >>>>> >> dist.apache.org <http://dist.apache.org> >>>>> >> > <http://dist.apache.org> >>>>> >> > >>>> <http://dist.apache.org> [2], >>>>> >> > >>>> which is signed with the key with fingerprint >>>>> C8282E76 [3], >>>>> >> > >>>> * all artifacts to be deployed to the Maven Central >>>>> Repository [4], >>>>> >> > >>>> * source code tag "v2.3.0-RC1" [5], >>>>> >> > >>>> * website pull request listing the release and >>>>> publishing the API >>>>> >> reference >>>>> >> > >>>> manual [6]. >>>>> >> > >>>> * Java artifacts were built with Maven 3.3.9 and >>>>> Oracle JDK >>>>> >> 1.8.0_111. >>>>> >> > >>>> * Python artifacts are deployed along with the >>>>> source release to the >>>>> >> > >>>> dist.apache.org <http://dist.apache.org> < >>>>> http://dist.apache.org> >>>>> >> <http://dist.apache.org> [2]. >>>>> >> > >>>> >>>>> >> > >>>> The vote will be open for at least 72 hours. It is >>>>> adopted by majority approval, >>>>> >> > >>>> with at least 3 PMC affirmative votes. >>>>> >> > >>>> >>>>> >> > >>>> Thanks, >>>>> >> > >>>> JB >>>>> >> > >>>> >>>>> >> > >>>> [1] >>>>> >> > >>>> https://issues.apache.org/ >>>>> jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341608 >>>>> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa? >>>>> projectId=12319527&version=12341608> >>>>> >> > <https://issues.apache.org/jira/secure/ReleaseNote.jspa? >>>>> projectId=12319527&version=12341608 >>>>> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa? >>>>> projectId=12319527&version=12341608>> >>>>> >> > >>>> [2] https://dist.apache.org/repos/ >>>>> dist/dev/beam/2.3.0/ >>>>> >> <https://dist.apache.org/repos/dist/dev/beam/2.3.0/> >>>>> >> > <https://dist.apache.org/repos/dist/dev/beam/2.3.0/ >>>>> >> <https://dist.apache.org/repos/dist/dev/beam/2.3.0/>> >>>>> >> > >>>> [3] https://dist.apache.org/repos/ >>>>> dist/release/beam/KEYS >>>>> >> <https://dist.apache.org/repos/dist/release/beam/KEYS> >>>>> >> > <https://dist.apache.org/repos/dist/release/beam/KEYS >>>>> >> <https://dist.apache.org/repos/dist/release/beam/KEYS>> >>>>> >> > >>>> [4] https://repository.apache.org/ >>>>> content/repositories/orgapachebeam-1026/ >>>>> >> <https://repository.apache.org/content/repositories/ >>>>> orgapachebeam-1026/> >>>>> >> > <https://repository.apache.org/content/repositories/ >>>>> orgapachebeam-1026/ >>>>> >> <https://repository.apache.org/content/repositories/ >>>>> orgapachebeam-1026/>> >>>>> >> > >>>> [5] https://github.com/apache/beam/tree/v2.3.0-RC1 >>>>> >> <https://github.com/apache/beam/tree/v2.3.0-RC1> >>>>> >> > <https://github.com/apache/beam/tree/v2.3.0-RC1 >>>>> >> <https://github.com/apache/beam/tree/v2.3.0-RC1>> >>>>> >> > >>>> [6] https://github.com/apache/beam-site/pull/381 >>>>> >> <https://github.com/apache/beam-site/pull/381> >>>>> >> > <https://github.com/apache/beam-site/pull/381 >>>>> >> <https://github.com/apache/beam-site/pull/381>> >>>>> >> > >>>> >>>>> >> > >>> >>>>> >> > >> >>>>> >> > >> -- >>>>> >> > >> Jean-Baptiste Onofré >>>>> >> > >> [email protected] <mailto:[email protected]> >>>>> <mailto:[email protected] >>>>> >> <mailto:[email protected]>> >>>>> >> > <mailto:[email protected] <mailto:[email protected]> >>>>> >> <mailto:[email protected] <mailto:[email protected]>>> >>>>> >> > >> http://blog.nanthrax.net >>>>> >> > >> Talend - http://www.talend.com >>>>> >> > > >>>>> >> > >>>>> >> > -- >>>>> >> > Jean-Baptiste Onofré >>>>> >> > [email protected] <mailto:[email protected]> >>>>> >> <mailto:[email protected] <mailto:[email protected]>> >>>>> >> > http://blog.nanthrax.net >>>>> >> > Talend - http://www.talend.com >>>>> >> > >>>>> >> > >>>>> >> >>>>> >> -- >>>>> >> Jean-Baptiste Onofré >>>>> >> [email protected] <mailto:[email protected]> >>>>> >> http://blog.nanthrax.net >>>>> >> Talend - http://www.talend.com >>>>> >> >>>>> >> >>>>> > >>>>> >>>>> -- >>>>> Jean-Baptiste Onofré >>>>> [email protected] >>>>> http://blog.nanthrax.net >>>>> Talend - http://www.talend.com >>>>> >>>> >>>> >>> >>
