+1 for the rollback. On Tue, Feb 6, 2018 at 9:24 AM, Kenneth Knowles <[email protected]> wrote:
> +1 for option 1. It is almost certainly repeated proto parsing. That is > fixed for ParDo via generalized caching but not for other transforms. Flink > also has a proto round trip, but the design might avoid the problem anyhow. > We should investigate before releasing, or just preemptively roll it back > too. The proto round trips are helpful for maturing portability and for > preventing hacks from returning but otherwise don't matter so much yet. > > Kenn > > On Tue, Feb 6, 2018 at 8:54 AM, Jean-Baptiste Onofré <[email protected]> > wrote: > >> Hi all, >> >> As you might have seen, I found the cause of performance degradation on >> the >> direct runner (BEAM-3617). >> >> We have basically three options for RC2: >> >> 1. We revert the change for 2.3.0. I think it's the fastest and more >> secure way. >> I created a PR for that (https://github.com/apache/beam/pull/4609). It >> gives us >> time to investigate and provide a better fix in Runner API for 2.4.0. >> >> 2. If the fix is easy and with limited impact in Runner API, we do that. >> >> 3. We leave as it is, adding a note that direct runner has worse >> performance in >> 2.3.0 compared to 2.2.0. >> >> In any case, I would like to cut RC2 later tonight or tomorrow morning >> (my time). >> >> Thoughts ? >> >> Regards >> JB >> >> On 02/06/2018 09:42 AM, Jean-Baptiste Onofré wrote: >> > Hi Reuven, >> > >> > it's what I'm suspecting. git bisect should give us more information >> (still in >> > progress, 3 more steps to complete ;)). I keep you posted. >> > >> > Regards >> > JB >> > >> > On 02/06/2018 08:36 AM, Reuven Lax wrote: >> >> Could this be related to any of the portability changes? >> >> >> >> On Mon, Feb 5, 2018 at 7:51 AM, Jean-Baptiste Onofré <[email protected] >> >> <mailto:[email protected]>> wrote: >> >> >> >> Created: >> >> >> >> https://issues.apache.org/jira/browse/BEAM-3617 >> >> <https://issues.apache.org/jira/browse/BEAM-3617> >> >> >> >> Regards >> >> JB >> >> >> >> On 02/05/2018 04:42 PM, Kenneth Knowles wrote: >> >> > What is the Jira for direct runner perf? >> >> > >> >> > On Mon, Feb 5, 2018 at 4:35 AM, Jean-Baptiste Onofré < >> [email protected] <mailto:[email protected]> >> >> > <mailto:[email protected] <mailto:[email protected]>>> wrote: >> >> > >> >> > Thanks ! >> >> > >> >> > I cherry-pick on release-2.3.0 branch. >> >> > >> >> > I'm on the direct runner perf test in the mean time. >> >> > >> >> > Thanks again ! >> >> > >> >> > Regards >> >> > JB >> >> > >> >> > On 02/05/2018 12:06 PM, Aljoscha Krettek wrote: >> >> > > I merged fixes for: >> >> > > - https://issues.apache.org/jira/browse/BEAM-3186 >> >> <https://issues.apache.org/jira/browse/BEAM-3186> >> >> > <https://issues.apache.org/jira/browse/BEAM-3186 >> >> <https://issues.apache.org/jira/browse/BEAM-3186>> >> >> > > - https://issues.apache.org/jira/browse/BEAM-3589 >> >> <https://issues.apache.org/jira/browse/BEAM-3589> >> >> > <https://issues.apache.org/jira/browse/BEAM-3589 >> >> <https://issues.apache.org/jira/browse/BEAM-3589>> >> >> > > >> >> > > @JB I didn't yet merge them on the 2.3.0 branch, though, >> but I can or you can go >> >> > > ahead. >> >> > > >> >> > >> On 5. Feb 2018, at 06:30, Jean-Baptiste Onofré < >> [email protected] <mailto:[email protected]> <mailto:[email protected] >> >> <mailto:[email protected]>> >> >> > >> <mailto:[email protected] <mailto:[email protected]> >> >> <mailto:[email protected] <mailto:[email protected]>>>> wrote: >> >> > >> >> >> > >> Hi guys, >> >> > >> >> >> > >> Quick update on the RC2 preparation: >> >> > >> >> >> > >> * BEAM-3587 (TextIO with Flink) seems related to a custom >> build with >> >> > Gradle (not >> >> > >> using artifacts created by Maven). Anyway, I will take a >> look today. >> >> > >> * BEAM-3186 has a PR. Aljoscha will do the review pretty >> soon. >> >> > >> * I'm also taking a look on the direct runner performance >> change. >> >> > >> >> >> > >> The goal is to cut RC2 tomorrow or the day after. >> >> > >> >> >> > >> Regards >> >> > >> JB >> >> > >> >> >> > >> On 02/01/2018 04:16 PM, Jean-Baptiste Onofré wrote: >> >> > >>> Hi guys, >> >> > >>> >> >> > >>> Especially due to BEAM-3587 & BEAM-3186 regressions, I >> cancel RC1. >> >> > >>> >> >> > >>> We will cherry-pick fixes on release-2.3.0 branch. >> >> > >>> >> >> > >>> I'm updating Jira right now. When the fixes will be >> cherry-picked, I >> >> > will submit >> >> > >>> a RC2 to vote. >> >> > >>> >> >> > >>> Thanks ! >> >> > >>> Regards >> >> > >>> JB >> >> > >>> >> >> > >>> On 01/30/2018 09:04 AM, Jean-Baptiste Onofré wrote: >> >> > >>>> Hi everyone, >> >> > >>>> >> >> > >>>> Please review and vote on the release candidate #1 for >> the version >> >> > 2.3.0, as >> >> > >>>> follows: >> >> > >>>> >> >> > >>>> [ ] +1, Approve the release >> >> > >>>> [ ] -1, Do not approve the release (please provide >> specific comments) >> >> > >>>> >> >> > >>>> >> >> > >>>> The complete staging area is available for your review, >> which >> >> includes: >> >> > >>>> * JIRA release notes [1], >> >> > >>>> * the official Apache source release to be deployed to >> >> dist.apache.org <http://dist.apache.org> >> >> > <http://dist.apache.org> >> >> > >>>> <http://dist.apache.org> [2], >> >> > >>>> which is signed with the key with fingerprint C8282E76 >> [3], >> >> > >>>> * all artifacts to be deployed to the Maven Central >> Repository [4], >> >> > >>>> * source code tag "v2.3.0-RC1" [5], >> >> > >>>> * website pull request listing the release and >> publishing the API >> >> reference >> >> > >>>> manual [6]. >> >> > >>>> * Java artifacts were built with Maven 3.3.9 and Oracle >> JDK >> >> 1.8.0_111. >> >> > >>>> * Python artifacts are deployed along with the source >> release to the >> >> > >>>> dist.apache.org <http://dist.apache.org> < >> http://dist.apache.org> >> >> <http://dist.apache.org> [2]. >> >> > >>>> >> >> > >>>> The vote will be open for at least 72 hours. It is >> adopted by majority approval, >> >> > >>>> with at least 3 PMC affirmative votes. >> >> > >>>> >> >> > >>>> Thanks, >> >> > >>>> JB >> >> > >>>> >> >> > >>>> [1] >> >> > >>>> https://issues.apache.org/jira >> /secure/ReleaseNote.jspa?projectId=12319527&version=12341608 >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?pro >> jectId=12319527&version=12341608> >> >> > <https://issues.apache.org/jira/secure/ReleaseNote.jspa?pro >> jectId=12319527&version=12341608 >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?pro >> jectId=12319527&version=12341608>> >> >> > >>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.3.0/ >> >> <https://dist.apache.org/repos/dist/dev/beam/2.3.0/> >> >> > <https://dist.apache.org/repos/dist/dev/beam/2.3.0/ >> >> <https://dist.apache.org/repos/dist/dev/beam/2.3.0/>> >> >> > >>>> [3] https://dist.apache.org/repos/ >> dist/release/beam/KEYS >> >> <https://dist.apache.org/repos/dist/release/beam/KEYS> >> >> > <https://dist.apache.org/repos/dist/release/beam/KEYS >> >> <https://dist.apache.org/repos/dist/release/beam/KEYS>> >> >> > >>>> [4] https://repository.apache.org/ >> content/repositories/orgapachebeam-1026/ >> >> <https://repository.apache.org/content/repositories/orgapac >> hebeam-1026/> >> >> > <https://repository.apache.org/content/repositories/orgapac >> hebeam-1026/ >> >> <https://repository.apache.org/content/repositories/orgapac >> hebeam-1026/>> >> >> > >>>> [5] https://github.com/apache/beam/tree/v2.3.0-RC1 >> >> <https://github.com/apache/beam/tree/v2.3.0-RC1> >> >> > <https://github.com/apache/beam/tree/v2.3.0-RC1 >> >> <https://github.com/apache/beam/tree/v2.3.0-RC1>> >> >> > >>>> [6] https://github.com/apache/beam-site/pull/381 >> >> <https://github.com/apache/beam-site/pull/381> >> >> > <https://github.com/apache/beam-site/pull/381 >> >> <https://github.com/apache/beam-site/pull/381>> >> >> > >>>> >> >> > >>> >> >> > >> >> >> > >> -- >> >> > >> Jean-Baptiste Onofré >> >> > >> [email protected] <mailto:[email protected]> <mailto: >> [email protected] >> >> <mailto:[email protected]>> >> >> > <mailto:[email protected] <mailto:[email protected]> >> >> <mailto:[email protected] <mailto:[email protected]>>> >> >> > >> http://blog.nanthrax.net >> >> > >> Talend - http://www.talend.com >> >> > > >> >> > >> >> > -- >> >> > Jean-Baptiste Onofré >> >> > [email protected] <mailto:[email protected]> >> >> <mailto:[email protected] <mailto:[email protected]>> >> >> > http://blog.nanthrax.net >> >> > Talend - http://www.talend.com >> >> > >> >> > >> >> >> >> -- >> >> Jean-Baptiste Onofré >> >> [email protected] <mailto:[email protected]> >> >> http://blog.nanthrax.net >> >> Talend - http://www.talend.com >> >> >> >> >> > >> >> -- >> Jean-Baptiste Onofré >> [email protected] >> http://blog.nanthrax.net >> Talend - http://www.talend.com >> > >
