Hi all, As you might have seen, I found the cause of performance degradation on the direct runner (BEAM-3617).
We have basically three options for RC2: 1. We revert the change for 2.3.0. I think it's the fastest and more secure way. I created a PR for that (https://github.com/apache/beam/pull/4609). It gives us time to investigate and provide a better fix in Runner API for 2.4.0. 2. If the fix is easy and with limited impact in Runner API, we do that. 3. We leave as it is, adding a note that direct runner has worse performance in 2.3.0 compared to 2.2.0. In any case, I would like to cut RC2 later tonight or tomorrow morning (my time). Thoughts ? Regards JB On 02/06/2018 09:42 AM, Jean-Baptiste Onofré wrote: > Hi Reuven, > > it's what I'm suspecting. git bisect should give us more information (still in > progress, 3 more steps to complete ;)). I keep you posted. > > Regards > JB > > On 02/06/2018 08:36 AM, Reuven Lax wrote: >> Could this be related to any of the portability changes? >> >> On Mon, Feb 5, 2018 at 7:51 AM, Jean-Baptiste Onofré <j...@nanthrax.net >> <mailto:j...@nanthrax.net>> wrote: >> >> Created: >> >> https://issues.apache.org/jira/browse/BEAM-3617 >> <https://issues.apache.org/jira/browse/BEAM-3617> >> >> Regards >> JB >> >> On 02/05/2018 04:42 PM, Kenneth Knowles wrote: >> > What is the Jira for direct runner perf? >> > >> > On Mon, Feb 5, 2018 at 4:35 AM, Jean-Baptiste Onofré >> <j...@nanthrax.net <mailto:j...@nanthrax.net> >> > <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net>>> wrote: >> > >> > Thanks ! >> > >> > I cherry-pick on release-2.3.0 branch. >> > >> > I'm on the direct runner perf test in the mean time. >> > >> > Thanks again ! >> > >> > Regards >> > JB >> > >> > On 02/05/2018 12:06 PM, Aljoscha Krettek wrote: >> > > I merged fixes for: >> > > - https://issues.apache.org/jira/browse/BEAM-3186 >> <https://issues.apache.org/jira/browse/BEAM-3186> >> > <https://issues.apache.org/jira/browse/BEAM-3186 >> <https://issues.apache.org/jira/browse/BEAM-3186>> >> > > - https://issues.apache.org/jira/browse/BEAM-3589 >> <https://issues.apache.org/jira/browse/BEAM-3589> >> > <https://issues.apache.org/jira/browse/BEAM-3589 >> <https://issues.apache.org/jira/browse/BEAM-3589>> >> > > >> > > @JB I didn't yet merge them on the 2.3.0 branch, though, but I >> can or you can go >> > > ahead. >> > > >> > >> On 5. Feb 2018, at 06:30, Jean-Baptiste Onofré >> <j...@nanthrax.net <mailto:j...@nanthrax.net> <mailto:j...@nanthrax.net >> <mailto:j...@nanthrax.net>> >> > >> <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net> >> <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net>>>> wrote: >> > >> >> > >> Hi guys, >> > >> >> > >> Quick update on the RC2 preparation: >> > >> >> > >> * BEAM-3587 (TextIO with Flink) seems related to a custom build >> with >> > Gradle (not >> > >> using artifacts created by Maven). Anyway, I will take a look >> today. >> > >> * BEAM-3186 has a PR. Aljoscha will do the review pretty soon. >> > >> * I'm also taking a look on the direct runner performance >> change. >> > >> >> > >> The goal is to cut RC2 tomorrow or the day after. >> > >> >> > >> Regards >> > >> JB >> > >> >> > >> On 02/01/2018 04:16 PM, Jean-Baptiste Onofré wrote: >> > >>> Hi guys, >> > >>> >> > >>> Especially due to BEAM-3587 & BEAM-3186 regressions, I cancel >> RC1. >> > >>> >> > >>> We will cherry-pick fixes on release-2.3.0 branch. >> > >>> >> > >>> I'm updating Jira right now. When the fixes will be >> cherry-picked, I >> > will submit >> > >>> a RC2 to vote. >> > >>> >> > >>> Thanks ! >> > >>> Regards >> > >>> JB >> > >>> >> > >>> On 01/30/2018 09:04 AM, Jean-Baptiste Onofré wrote: >> > >>>> Hi everyone, >> > >>>> >> > >>>> Please review and vote on the release candidate #1 for the >> version >> > 2.3.0, as >> > >>>> follows: >> > >>>> >> > >>>> [ ] +1, Approve the release >> > >>>> [ ] -1, Do not approve the release (please provide specific >> comments) >> > >>>> >> > >>>> >> > >>>> The complete staging area is available for your review, which >> includes: >> > >>>> * JIRA release notes [1], >> > >>>> * the official Apache source release to be deployed to >> dist.apache.org <http://dist.apache.org> >> > <http://dist.apache.org> >> > >>>> <http://dist.apache.org> [2], >> > >>>> which is signed with the key with fingerprint C8282E76 [3], >> > >>>> * all artifacts to be deployed to the Maven Central >> Repository [4], >> > >>>> * source code tag "v2.3.0-RC1" [5], >> > >>>> * website pull request listing the release and publishing the >> API >> reference >> > >>>> manual [6]. >> > >>>> * Java artifacts were built with Maven 3.3.9 and Oracle JDK >> 1.8.0_111. >> > >>>> * Python artifacts are deployed along with the source release >> to the >> > >>>> dist.apache.org <http://dist.apache.org> >> <http://dist.apache.org> >> <http://dist.apache.org> [2]. >> > >>>> >> > >>>> The vote will be open for at least 72 hours. It is adopted by >> majority approval, >> > >>>> with at least 3 PMC affirmative votes. >> > >>>> >> > >>>> Thanks, >> > >>>> JB >> > >>>> >> > >>>> [1] >> > >>>> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341608 >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341608> >> > >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341608 >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341608>> >> > >>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.3.0/ >> <https://dist.apache.org/repos/dist/dev/beam/2.3.0/> >> > <https://dist.apache.org/repos/dist/dev/beam/2.3.0/ >> <https://dist.apache.org/repos/dist/dev/beam/2.3.0/>> >> > >>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS >> <https://dist.apache.org/repos/dist/release/beam/KEYS> >> > <https://dist.apache.org/repos/dist/release/beam/KEYS >> <https://dist.apache.org/repos/dist/release/beam/KEYS>> >> > >>>> [4] >> https://repository.apache.org/content/repositories/orgapachebeam-1026/ >> <https://repository.apache.org/content/repositories/orgapachebeam-1026/> >> > >> <https://repository.apache.org/content/repositories/orgapachebeam-1026/ >> <https://repository.apache.org/content/repositories/orgapachebeam-1026/>> >> > >>>> [5] https://github.com/apache/beam/tree/v2.3.0-RC1 >> <https://github.com/apache/beam/tree/v2.3.0-RC1> >> > <https://github.com/apache/beam/tree/v2.3.0-RC1 >> <https://github.com/apache/beam/tree/v2.3.0-RC1>> >> > >>>> [6] https://github.com/apache/beam-site/pull/381 >> <https://github.com/apache/beam-site/pull/381> >> > <https://github.com/apache/beam-site/pull/381 >> <https://github.com/apache/beam-site/pull/381>> >> > >>>> >> > >>> >> > >> >> > >> -- >> > >> Jean-Baptiste Onofré >> > >> jbono...@apache.org <mailto:jbono...@apache.org> >> <mailto:jbono...@apache.org >> <mailto:jbono...@apache.org>> >> > <mailto:jbono...@apache.org <mailto:jbono...@apache.org> >> <mailto:jbono...@apache.org <mailto:jbono...@apache.org>>> >> > >> http://blog.nanthrax.net >> > >> Talend - http://www.talend.com >> > > >> > >> > -- >> > Jean-Baptiste Onofré >> > jbono...@apache.org <mailto:jbono...@apache.org> >> <mailto:jbono...@apache.org <mailto:jbono...@apache.org>> >> > http://blog.nanthrax.net >> > Talend - http://www.talend.com >> > >> > >> >> -- >> Jean-Baptiste Onofré >> jbono...@apache.org <mailto:jbono...@apache.org> >> http://blog.nanthrax.net >> Talend - http://www.talend.com >> >> > -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com
