If https://issues.apache.org/jira/browse/BEAM-2671 is a 2.1.0 blocker then https://issues.apache.org/jira/browse/BEAM-1868 also should be, because it's a failure of another method in the same test and I suppose it indicates brokenness to the same extent. Or both shouldn't.
Given the progress so far, the chances of resolving the JIRA quickly are looking bleak to me now, and the release has been going on for almost 1 month, and many large improvements have been added to Beam HEAD since the first RC was cut. I'm still in favor of: 1) cutting 2.1.0 RC3 immediately, and acknowledging that streaming in Spark runner in cluster mode is still (potentially) broken in this release - to the same or smaller extent than in 2.0.0, so this is not a regression. The extent is still not clear to me; I asked on the JIRA. 2) immediately or very soon after this 2.1.0, start cutting 2.2.0, and target these issues to 2.2.0. My argument is: - 2.1.0 contains 2.5 months worth of new features, and releasing them will benefit a lot of existing Beam users - I don't think there are that many users for whom it's critically important whether the first release with working Spark streaming will be 2.1.0 or 2.2.0, especially if we start cutting 2.2.0 very soon. This is speculation though - (subjective personal feeling) The release process requires participation and momentum from community members, and letting it drag on for too long loses that momentum. We should anyway pursue resolving the issues asap, and users who were eagerly waiting for Spark streaming to work properly can run Beam at HEAD in the window between when they are first resolved and when 2.2.0 is released. What do you think? On Sat, Aug 5, 2017 at 9:31 PM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > Another quick update. > > Aviem updated the Jira as he and his team wants to take a look. I'm also > doing a > new bisect on my side. I've given an extra day to move forward. If we > don't have > clear statement tonight, then, I will cut the RC3 tonight or tomorrow > morning > (my time). > > Regards > JB > > On 08/05/2017 02:37 AM, Eugene Kirpichov wrote: > > I did some more investigation on that JIRA > > https://issues.apache.org/jira/browse/BEAM-2671 and my conclusion is: > > > > We need to postpone that JIRA to 2.2.0 and finalize release 2.1.0 as-is. > > > > The TL;DR of my investigation is that: > > - We have some confidence that Spark runner in 2.1.0 generally works > > properly: it passes ValidatesRunner tests, and there's been some amount > of > > manual testing. > > - Release 2.0.0 does not contain a critical fix and, if I understand > > correctly, Spark runner at 2.0.0 was basically unusable in streaming > > cluster mode. > > - So, even if the JIRA signals that there is something wrong in the Spark > > runner at 2.1.0, it's definitely better than 2.0.0 so there is no > > regression for the user. > > > > I moved the JIRA to 2.2.0 so there are no blocking issues remaining for > > 2.1.0. JB - the next step is for you to proceed with cutting the RC, > > correct? > > > > Thanks. > > > > On Thu, Aug 3, 2017 at 7:04 AM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > > > >> Another quick update. Regarding BEAM-2671, I asked help from Stas and > >> Aviem on > >> this one. It's our high priority as it's the main blocking issue before > >> cutting RC3. > >> > >> At some point, if we are not able to move fast on this one, I would > >> propose to > >> cut RC3 as it is. > >> > >> Regards > >> JB > >> > >> On 08/02/2017 08:52 PM, Jean-Baptiste Onofré wrote: > >>> Hi, > >>> > >>> Thanks Eugene for the sumup. > >>> > >>> BEAM-2708 is now fixed. > >>> > >>> The last blocking issue for RC3 is BEAM-2671. I spent time today on > this > >> one, > >>> investigating the different issues. > >>> > >>> Agree that help from Aviem and Kenn would help for sure. > >>> > >>> Aviem already started to kindly take a look on the Jira today. > >>> > >>> Clearly, it would be great to fix BEAM-2671 in the coming 36 hours. I > >> would like > >>> to submit RC3 to vote tomorrow or the day after (my time). > >>> > >>> Thanks ! > >>> Regards > >>> JB > >>> > >>> On 08/02/2017 08:24 PM, Eugene Kirpichov wrote: > >>>> We're down to 2 issues. > >>>> > >>>> BEAM-2670 has been fixed. > >>>> https://issues.apache.org/jira/browse/BEAM-2708 has a fix in review > >>>> > >>>> https://issues.apache.org/jira/browse/BEAM-2671 is the nasty one and > we > >>>> don't understand it nor have a fix. Help is needed; some people who > >> could > >>>> help are +Kenn Knowles <k...@google.com> and +Aviem Zur < > >> aviem...@gmail.com> > >>>> . > >>>> > >>>> On Thu, Jul 27, 2017 at 6:41 AM Jean-Baptiste Onofré <j...@nanthrax.net > > > >>>> wrote: > >>>> > >>>>> Hi guys, > >>>>> > >>>>> We have three open issues for the 2.1.0 that we need to fix before I > >> will > >>>>> be > >>>>> able to cut RC3: > >>>>> > >>>>> https://issues.apache.org/jira/projects/BEAM/versions/12340528 > >>>>> > >>>>> I'm working on BEAM-2671. > >>>>> > >>>>> Any help is welcome for the two other Jiras (BEAM-2587 and > BEAM-2670). > >>>>> > >>>>> Thanks ! > >>>>> Regards > >>>>> JB > >>>>> > >>>>> On 07/18/2017 06:30 PM, Jean-Baptiste Onofré wrote: > >>>>>> Hi everyone, > >>>>>> > >>>>>> Please review and vote on the release candidate #2 for the version > >>>>> 2.1.0, as > >>>>>> follows: > >>>>>> > >>>>>> [ ] +1, Approve the release > >>>>>> [ ] -1, Do not approve the release (please provide specific > comments) > >>>>>> > >>>>>> > >>>>>> The complete staging area is available for your review, which > >> includes: > >>>>>> * JIRA release notes [1], > >>>>>> * the official Apache source release to be deployed to > >> dist.apache.org > >>>>> [2], > >>>>>> which is signed with the key with fingerprint C8282E76 [3], > >>>>>> * all artifacts to be deployed to the Maven Central Repository [4], > >>>>>> * source code tag "v2.1.0-RC2" [5], > >>>>>> * website pull request listing the release and publishing the API > >>>>> reference > >>>>>> manual [6]. > >>>>>> * Python artifacts are deployed along with the source release to the > >>>>>> dist.apache.org [2]. > >>>>>> > >>>>>> The vote will be open for at least 72 hours. It is adopted by > majority > >>>>> approval, > >>>>>> with at least 3 PMC affirmative votes. > >>>>>> > >>>>>> Thanks, > >>>>>> JB > >>>>>> > >>>>>> [1] > >>>>>> > >>>>> > >> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12340528 > >>>>> > >>>>>> > >>>>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.1.0/ > >>>>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS > >>>>>> [4] > >>>>> > https://repository.apache.org/content/repositories/orgapachebeam-1019/ > >>>>>> [5] https://github.com/apache/beam/tree/v2.1.0-RC2 > >>>>>> [6] https://github.com/apache/beam-site/pull/270 > >>>>> > >>>>> -- > >>>>> Jean-Baptiste Onofré > >>>>> jbono...@apache.org > >>>>> http://blog.nanthrax.net > >>>>> Talend - http://www.talend.com > >>>>> > >>>> > >>> > >> > >> -- > >> Jean-Baptiste Onofré > >> jbono...@apache.org > >> http://blog.nanthrax.net > >> Talend - http://www.talend.com > >> > > > > -- > Jean-Baptiste Onofré > jbono...@apache.org > http://blog.nanthrax.net > Talend - http://www.talend.com >
