+1 for rollback as well. On Tue, Feb 6, 2018 at 12:37 PM, Reuven Lax <[email protected]> wrote:
> +1 for the rollback. > > On Tue, Feb 6, 2018 at 9:24 AM, Kenneth Knowles <[email protected]> wrote: > >> +1 for option 1. It is almost certainly repeated proto parsing. That is >> fixed for ParDo via generalized caching but not for other transforms. Flink >> also has a proto round trip, but the design might avoid the problem anyhow. >> We should investigate before releasing, or just preemptively roll it back >> too. The proto round trips are helpful for maturing portability and for >> preventing hacks from returning but otherwise don't matter so much yet. >> >> Kenn >> >> On Tue, Feb 6, 2018 at 8:54 AM, Jean-Baptiste Onofré <[email protected]> >> wrote: >> >>> Hi all, >>> >>> As you might have seen, I found the cause of performance degradation on >>> the >>> direct runner (BEAM-3617). >>> >>> We have basically three options for RC2: >>> >>> 1. We revert the change for 2.3.0. I think it's the fastest and more >>> secure way. >>> I created a PR for that (https://github.com/apache/beam/pull/4609). It >>> gives us >>> time to investigate and provide a better fix in Runner API for 2.4.0. >>> >>> 2. If the fix is easy and with limited impact in Runner API, we do that. >>> >>> 3. We leave as it is, adding a note that direct runner has worse >>> performance in >>> 2.3.0 compared to 2.2.0. >>> >>> In any case, I would like to cut RC2 later tonight or tomorrow morning >>> (my time). >>> >>> Thoughts ? >>> >>> Regards >>> JB >>> >>> On 02/06/2018 09:42 AM, Jean-Baptiste Onofré wrote: >>> > Hi Reuven, >>> > >>> > it's what I'm suspecting. git bisect should give us more information >>> (still in >>> > progress, 3 more steps to complete ;)). I keep you posted. >>> > >>> > Regards >>> > JB >>> > >>> > On 02/06/2018 08:36 AM, Reuven Lax wrote: >>> >> Could this be related to any of the portability changes? >>> >> >>> >> On Mon, Feb 5, 2018 at 7:51 AM, Jean-Baptiste Onofré <[email protected] >>> >> <mailto:[email protected]>> wrote: >>> >> >>> >> Created: >>> >> >>> >> https://issues.apache.org/jira/browse/BEAM-3617 >>> >> <https://issues.apache.org/jira/browse/BEAM-3617> >>> >> >>> >> Regards >>> >> JB >>> >> >>> >> On 02/05/2018 04:42 PM, Kenneth Knowles wrote: >>> >> > What is the Jira for direct runner perf? >>> >> > >>> >> > On Mon, Feb 5, 2018 at 4:35 AM, Jean-Baptiste Onofré < >>> [email protected] <mailto:[email protected]> >>> >> > <mailto:[email protected] <mailto:[email protected]>>> wrote: >>> >> > >>> >> > Thanks ! >>> >> > >>> >> > I cherry-pick on release-2.3.0 branch. >>> >> > >>> >> > I'm on the direct runner perf test in the mean time. >>> >> > >>> >> > Thanks again ! >>> >> > >>> >> > Regards >>> >> > JB >>> >> > >>> >> > On 02/05/2018 12:06 PM, Aljoscha Krettek wrote: >>> >> > > I merged fixes for: >>> >> > > - https://issues.apache.org/jira/browse/BEAM-3186 >>> >> <https://issues.apache.org/jira/browse/BEAM-3186> >>> >> > <https://issues.apache.org/jira/browse/BEAM-3186 >>> >> <https://issues.apache.org/jira/browse/BEAM-3186>> >>> >> > > - https://issues.apache.org/jira/browse/BEAM-3589 >>> >> <https://issues.apache.org/jira/browse/BEAM-3589> >>> >> > <https://issues.apache.org/jira/browse/BEAM-3589 >>> >> <https://issues.apache.org/jira/browse/BEAM-3589>> >>> >> > > >>> >> > > @JB I didn't yet merge them on the 2.3.0 branch, though, >>> but I can or you can go >>> >> > > ahead. >>> >> > > >>> >> > >> On 5. Feb 2018, at 06:30, Jean-Baptiste Onofré < >>> [email protected] <mailto:[email protected]> <mailto:[email protected] >>> >> <mailto:[email protected]>> >>> >> > >> <mailto:[email protected] <mailto:[email protected]> >>> >> <mailto:[email protected] <mailto:[email protected]>>>> wrote: >>> >> > >> >>> >> > >> Hi guys, >>> >> > >> >>> >> > >> Quick update on the RC2 preparation: >>> >> > >> >>> >> > >> * BEAM-3587 (TextIO with Flink) seems related to a >>> custom build with >>> >> > Gradle (not >>> >> > >> using artifacts created by Maven). Anyway, I will take a >>> look today. >>> >> > >> * BEAM-3186 has a PR. Aljoscha will do the review pretty >>> soon. >>> >> > >> * I'm also taking a look on the direct runner >>> performance change. >>> >> > >> >>> >> > >> The goal is to cut RC2 tomorrow or the day after. >>> >> > >> >>> >> > >> Regards >>> >> > >> JB >>> >> > >> >>> >> > >> On 02/01/2018 04:16 PM, Jean-Baptiste Onofré wrote: >>> >> > >>> Hi guys, >>> >> > >>> >>> >> > >>> Especially due to BEAM-3587 & BEAM-3186 regressions, I >>> cancel RC1. >>> >> > >>> >>> >> > >>> We will cherry-pick fixes on release-2.3.0 branch. >>> >> > >>> >>> >> > >>> I'm updating Jira right now. When the fixes will be >>> cherry-picked, I >>> >> > will submit >>> >> > >>> a RC2 to vote. >>> >> > >>> >>> >> > >>> Thanks ! >>> >> > >>> Regards >>> >> > >>> JB >>> >> > >>> >>> >> > >>> On 01/30/2018 09:04 AM, Jean-Baptiste Onofré wrote: >>> >> > >>>> Hi everyone, >>> >> > >>>> >>> >> > >>>> Please review and vote on the release candidate #1 for >>> the version >>> >> > 2.3.0, as >>> >> > >>>> follows: >>> >> > >>>> >>> >> > >>>> [ ] +1, Approve the release >>> >> > >>>> [ ] -1, Do not approve the release (please provide >>> specific comments) >>> >> > >>>> >>> >> > >>>> >>> >> > >>>> The complete staging area is available for your >>> review, which >>> >> includes: >>> >> > >>>> * JIRA release notes [1], >>> >> > >>>> * the official Apache source release to be deployed to >>> >> dist.apache.org <http://dist.apache.org> >>> >> > <http://dist.apache.org> >>> >> > >>>> <http://dist.apache.org> [2], >>> >> > >>>> which is signed with the key with fingerprint C8282E76 >>> [3], >>> >> > >>>> * all artifacts to be deployed to the Maven Central >>> Repository [4], >>> >> > >>>> * source code tag "v2.3.0-RC1" [5], >>> >> > >>>> * website pull request listing the release and >>> publishing the API >>> >> reference >>> >> > >>>> manual [6]. >>> >> > >>>> * Java artifacts were built with Maven 3.3.9 and >>> Oracle JDK >>> >> 1.8.0_111. >>> >> > >>>> * Python artifacts are deployed along with the source >>> release to the >>> >> > >>>> dist.apache.org <http://dist.apache.org> < >>> http://dist.apache.org> >>> >> <http://dist.apache.org> [2]. >>> >> > >>>> >>> >> > >>>> The vote will be open for at least 72 hours. It is >>> adopted by majority approval, >>> >> > >>>> with at least 3 PMC affirmative votes. >>> >> > >>>> >>> >> > >>>> Thanks, >>> >> > >>>> JB >>> >> > >>>> >>> >> > >>>> [1] >>> >> > >>>> https://issues.apache.org/jira >>> /secure/ReleaseNote.jspa?projectId=12319527&version=12341608 >>> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?pro >>> jectId=12319527&version=12341608> >>> >> > <https://issues.apache.org/jira/secure/ReleaseNote.jspa?pro >>> jectId=12319527&version=12341608 >>> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?pro >>> jectId=12319527&version=12341608>> >>> >> > >>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.3.0/ >>> >> <https://dist.apache.org/repos/dist/dev/beam/2.3.0/> >>> >> > <https://dist.apache.org/repos/dist/dev/beam/2.3.0/ >>> >> <https://dist.apache.org/repos/dist/dev/beam/2.3.0/>> >>> >> > >>>> [3] https://dist.apache.org/repos/ >>> dist/release/beam/KEYS >>> >> <https://dist.apache.org/repos/dist/release/beam/KEYS> >>> >> > <https://dist.apache.org/repos/dist/release/beam/KEYS >>> >> <https://dist.apache.org/repos/dist/release/beam/KEYS>> >>> >> > >>>> [4] https://repository.apache.org/ >>> content/repositories/orgapachebeam-1026/ >>> >> <https://repository.apache.org/content/repositories/orgapac >>> hebeam-1026/> >>> >> > <https://repository.apache.org/content/repositories/orgapac >>> hebeam-1026/ >>> >> <https://repository.apache.org/content/repositories/orgapac >>> hebeam-1026/>> >>> >> > >>>> [5] https://github.com/apache/beam/tree/v2.3.0-RC1 >>> >> <https://github.com/apache/beam/tree/v2.3.0-RC1> >>> >> > <https://github.com/apache/beam/tree/v2.3.0-RC1 >>> >> <https://github.com/apache/beam/tree/v2.3.0-RC1>> >>> >> > >>>> [6] https://github.com/apache/beam-site/pull/381 >>> >> <https://github.com/apache/beam-site/pull/381> >>> >> > <https://github.com/apache/beam-site/pull/381 >>> >> <https://github.com/apache/beam-site/pull/381>> >>> >> > >>>> >>> >> > >>> >>> >> > >> >>> >> > >> -- >>> >> > >> Jean-Baptiste Onofré >>> >> > >> [email protected] <mailto:[email protected]> >>> <mailto:[email protected] >>> >> <mailto:[email protected]>> >>> >> > <mailto:[email protected] <mailto:[email protected]> >>> >> <mailto:[email protected] <mailto:[email protected]>>> >>> >> > >> http://blog.nanthrax.net >>> >> > >> Talend - http://www.talend.com >>> >> > > >>> >> > >>> >> > -- >>> >> > Jean-Baptiste Onofré >>> >> > [email protected] <mailto:[email protected]> >>> >> <mailto:[email protected] <mailto:[email protected]>> >>> >> > http://blog.nanthrax.net >>> >> > Talend - http://www.talend.com >>> >> > >>> >> > >>> >> >>> >> -- >>> >> Jean-Baptiste Onofré >>> >> [email protected] <mailto:[email protected]> >>> >> http://blog.nanthrax.net >>> >> Talend - http://www.talend.com >>> >> >>> >> >>> > >>> >>> -- >>> Jean-Baptiste Onofré >>> [email protected] >>> http://blog.nanthrax.net >>> Talend - http://www.talend.com >>> >> >> >
