+1 to cherry-pick. It will make the release broken for enough users that we shouldn't produce a release without the fix.
Kenn On Sat, Apr 9, 2022 at 4:05 PM Ahmet Altay <al...@google.com> wrote: > +1 to cherry picking https://github.com/apache/beam/pull/17329 to the > release branch. > > For completeness, completing the cherry pick questionnaire: > - Is this a regression from a previous release? Yes > - Is this a new feature or related to a new feature? No > - Would this impact production workloads for users? Yes > - What percentage of users would be impacted by this issue if it is not > fixed? With GCS, probably a majority of Beam python users. > - Would it be possible for the impacted users to skip this version? Yes. > > Only the last answer does not strictly support a cherry pick request but > given the severity (potential data loss) and large number of impacted > production workloads, I think it will be the right action to cherry pick. > > > On Sat, Apr 9, 2022 at 10:33 AM Chamikara Jayalath <chamik...@google.com> > wrote: > >> Hi Daniel, >> >> Can we consider including https://github.com/apache/beam/pull/17329 in >> the ongoing release ? >> It fixes a potential data loss issue for Python gcsio and datastoreio >> users and the issue is a regression from a previous release so I think it >> meets cherry-pick criteria. >> >> Jira is https://issues.apache.org/jira/browse/BEAM-14282 >> >> Thanks, >> Cham >> >> On Wed, Apr 6, 2022 at 1:02 PM Ahmet Altay <al...@google.com> wrote: >> >>> Awesome! Thank you Daniel and everyone who helped on the blockers! >>> >>> On Wed, Apr 6, 2022 at 12:51 PM Daniel Oliveira <danolive...@google.com> >>> wrote: >>> >>>> Thanks everyone for helping look into these! I need to do a once-over >>>> to confirm but I think that clears every remaining bug to investigate, >>>> which means I can proceed on RC1. I'll keep this thread updated. >>>> >>>> On Wed, Apr 6, 2022 at 12:26 PM Kiley Sok <kiley...@google.com> wrote: >>>> >>>>> BEAM-14192 seems to have resolved itself. It's possible that the >>>>> internal changes just went out. >>>>> >>>>> On Wed, Apr 6, 2022 at 9:18 AM Ahmet Altay <al...@google.com> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Wed, Apr 6, 2022 at 7:13 AM Daniel Collins <dpcoll...@google.com> >>>>>> wrote: >>>>>> >>>>>>> https://issues.apache.org/jira/browse/BEAM-14253 is an internal >>>>>>> dataflow bug, see comments from luke there. >>>>>>> >>>>>> >>>>>> Thank you @Daniel Collins <dpcoll...@google.com> Luke's comment >>>>>> says the Dataflow fix is about to be rolled out. I would vote for not >>>>>> blocking the Beam release on this (and Dataflow rollout) and moving >>>>>> forward. My reason is that pubSublite IO currently has a very small user >>>>>> base, and the fix will be rolled in parallel independently from the Beam >>>>>> release. >>>>>> >>>>>> @Daniel Oliveira <danolive...@google.com> - Does it make sense? >>>>>> Could we continue with the release once other blockers are resolved? >>>>>> (IIUC >>>>>> https://issues.apache.org/jira/browse/BEAM-14192 would be the only >>>>>> remaining blocker.) >>>>>> >>>>>> >>>>>>> >>>>>>> On Tue, Apr 5, 2022 at 10:35 PM Ahmet Altay <al...@google.com> >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Apr 5, 2022 at 6:36 PM Daniel Oliveira < >>>>>>>> danolive...@google.com> wrote: >>>>>>>> >>>>>>>>> Quick update on the progress: Starting an RC ended up not >>>>>>>>> happening like I planned. The release-blocking bugs took a little >>>>>>>>> long to >>>>>>>>> wrap up, and then I had to triage a bunch of other test failures >>>>>>>>> (which are >>>>>>>>> logged in https://github.com/apache/beam/pull/17259). That's what >>>>>>>>> I've mainly been doing since the last update. >>>>>>>>> >>>>>>>> >>>>>>>> Thank you for doing that and thank you for the update Daniel! >>>>>>>> >>>>>>>> Please reach out in a few days if the blockers are still open >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> I think the biggest thing people can help with is helping triage >>>>>>>>> the sub-tasks in https://issues.apache.org/jira/browse/BEAM-14252. >>>>>>>>> Not necessarily fixing them, just triaging enough so I can identify >>>>>>>>> whether >>>>>>>>> they're release-blocking or not. These tasks are already assigned and >>>>>>>>> people are working on them, it's not as if they're being ignored, but >>>>>>>>> any >>>>>>>>> help that could get them triaged faster would do a lot to speed >>>>>>>>> things up. >>>>>>>>> >>>>>>>> >>>>>>>> There are 3 blockers: >>>>>>>> - https://issues.apache.org/jira/browse/BEAM-14222 - this seems to >>>>>>>> be fixed earlier today but JIRA has not been closed. >>>>>>>> - https://issues.apache.org/jira/browse/BEAM-14253 - It is about >>>>>>>> pubsub lite and assigned to @Daniel Collins <dpcoll...@google.com> >>>>>>>> . @Daniel Collins <dpcoll...@google.com> Could you please take a >>>>>>>> look? >>>>>>>> - https://issues.apache.org/jira/browse/BEAM-14192 - @Kiley Sok >>>>>>>> <kiley...@google.com> is working on this. (If you need help please >>>>>>>> reach out.) >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Apr 4, 2022 at 5:50 PM Ahmet Altay <al...@google.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Daniel, is there an update on the RC1's progress? Could we help? >>>>>>>>>> >>>>>>>>>> On Wed, Mar 30, 2022 at 7:56 PM Ahmet Altay <al...@google.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> It looks like BEAM-14163 is fixed. Thank you Robert for the fix. >>>>>>>>>>> Thank you Daniel for going through all benchmarks and noticing this! >>>>>>>>>>> >>>>>>>>>>> Daniel, what are the current release blockers? Do you need help? >>>>>>>>>>> >>>>>>>>>>> Thank you! >>>>>>>>>>> Ahmet >>>>>>>>>>> >>>>>>>>>>> On Tue, Mar 29, 2022 at 6:59 PM Ahmet Altay <al...@google.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Awesome, thanks for the update! >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Mar 29, 2022 at 6:09 PM Daniel Oliveira < >>>>>>>>>>>> danolive...@google.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Sorry for the lack of updates, I'll try to update this more >>>>>>>>>>>>> often. Valentyn was right about that performance regression being >>>>>>>>>>>>> the >>>>>>>>>>>>> biggest unknown. That and multiple other incoming cherry-picks >>>>>>>>>>>>> have been >>>>>>>>>>>>> coming in around today so I'm focusing on wrapping up as many >>>>>>>>>>>>> release-blockers as I can. It's looking likely that I can start >>>>>>>>>>>>> working on >>>>>>>>>>>>> an RC tomorrow. >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Mar 28, 2022 at 12:17 PM Robert Bradshaw < >>>>>>>>>>>>> rober...@google.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> I will follow up with a fix today. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, Mar 28, 2022 at 11:50 AM Ahmet Altay < >>>>>>>>>>>>>> al...@google.com> wrote: >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > @Daniel Oliveira - could you please continue with triaging >>>>>>>>>>>>>> existing blockers and running other release validation tests. >>>>>>>>>>>>>> Ideally we >>>>>>>>>>>>>> can catch and fix other issues in parallel. >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > On Mon, Mar 28, 2022 at 9:20 AM Valentyn Tymofieiev < >>>>>>>>>>>>>> valen...@google.com> wrote: >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> Daniel mentioned offline last week that >>>>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-14163 was the >>>>>>>>>>>>>> biggest unknown among the blocking bugs. I root caused it and >>>>>>>>>>>>>> passed to >>>>>>>>>>>>>> @Robert Bradshaw for the follow-up. >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > Thank you Valentyn! >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > @Robert Bradshaw - please let us know how we should proceed. >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> On Fri, Mar 25, 2022 at 5:03 PM Ahmet Altay < >>>>>>>>>>>>>> al...@google.com> wrote: >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> >>> Hi Daniel, >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> >>> How is the 2.38.0 release progressing? What are the >>>>>>>>>>>>>> remaining blockers? >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> >>> It would be good to move the release fast and make >>>>>>>>>>>>>> blocker/non-blocker decisions on the remaining issues early next >>>>>>>>>>>>>> week. >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> >>> Ahmet >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> >>> On Wed, Mar 23, 2022 at 6:24 PM Daniel Oliveira < >>>>>>>>>>>>>> danolive...@google.com> wrote: >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> >>>> Hello everyone, >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> >>>> I've cut the 2.38.0 release branch [1]. >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> >>>> Looking through release-blocking bugs [2] there are a >>>>>>>>>>>>>> few, but a fair amount of them I suspect aren't actual release >>>>>>>>>>>>>> blockers >>>>>>>>>>>>>> according to the criteria [3]. Starting tomorrow I'll go through >>>>>>>>>>>>>> the list >>>>>>>>>>>>>> and work with the owners to resolve them. >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> >>>> Finally, if you have any significant changes to >>>>>>>>>>>>>> announce, please send me a PR which adds them to CHANGES.md [4]. >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> >>>> Thanks, >>>>>>>>>>>>>> >>>> Daniel Oliveira >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> >>>> [1] https://github.com/apache/beam/tree/release-2.38.0 >>>>>>>>>>>>>> >>>> [2] >>>>>>>>>>>>>> https://issues.apache.org/jira/issues/?filter=-1&jql=project%3DBEAM%20and%20fixVersion%3D2.38.0%20%20AND%20statusCategory%20!%3D%20Done%20order%20by%20updated%20DESC >>>>>>>>>>>>>> >>>> [3] https://beam.apache.org/contribute/release-blocking/ >>>>>>>>>>>>>> >>>> [4] >>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/CHANGES.md >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> >>>> On Mon, Mar 14, 2022 at 3:47 PM Ahmet Altay < >>>>>>>>>>>>>> al...@google.com> wrote: >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> +1, thank you Daniel. >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> I hope the process has improved for the better since >>>>>>>>>>>>>> your last release. I would be very interested in a retrospective >>>>>>>>>>>>>> at the end >>>>>>>>>>>>>> of this release and learning from your experience. >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> On Mon, Mar 14, 2022 at 12:50 PM Kyle Weaver < >>>>>>>>>>>>>> kcwea...@google.com> wrote: >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> >>>>>> +1 thanks for taking this on Daniel! >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> >>>>>> On Thu, Mar 10, 2022 at 11:24 AM Daniel Oliveira < >>>>>>>>>>>>>> danolive...@google.com> wrote: >>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>> >>>>>>> That's the boilerplate email that everyone sends, but >>>>>>>>>>>>>> on a more personal note, this would be the second release I'm >>>>>>>>>>>>>> working on (I >>>>>>>>>>>>>> last worked on 2.24.0) so I'm interested to see how the release >>>>>>>>>>>>>> process has >>>>>>>>>>>>>> evolved. >>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>> >>>>>>> Thanks, >>>>>>>>>>>>>> >>>>>>> Daniel Oliveira >>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>> >>>>>>> On Thu, Mar 10, 2022 at 11:22 AM Daniel Oliveira < >>>>>>>>>>>>>> danolive...@google.com> wrote: >>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>> >>>>>>>> Hi all, >>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>> >>>>>>>> The next (2.38.0) release branch cut is scheduled >>>>>>>>>>>>>> for 23rd of March, according to >>>>>>>>>>>>>> >>>>>>>> the release calendar [1]. >>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>> >>>>>>>> I would like to volunteer myself to do this release. >>>>>>>>>>>>>> My plan is to cut the branch on that date, and cherrypick >>>>>>>>>>>>>> release-blocking >>>>>>>>>>>>>> fixes afterwards if any. >>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>> >>>>>>>> Please help the release go smoothly by: >>>>>>>>>>>>>> >>>>>>>> - Making sure that any unresolved release blocking >>>>>>>>>>>>>> JIRA issues for 2.38.0 should have their "Fix Version/s" marked >>>>>>>>>>>>>> as "2.38.0" >>>>>>>>>>>>>> as soon as possible. >>>>>>>>>>>>>> >>>>>>>> - Reviewing the current release blockers [2] and >>>>>>>>>>>>>> remove the "Fix Version/s" if they don't meet the criteria at >>>>>>>>>>>>>> [3]. >>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>> >>>>>>>> Let me know if you have any >>>>>>>>>>>>>> comments/objections/questions. >>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>> >>>>>>>> Thanks, >>>>>>>>>>>>>> >>>>>>>> Daniel Oliveira >>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>> >>>>>>>> [1] >>>>>>>>>>>>>> https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com >>>>>>>>>>>>>> >>>>>>>> [2] >>>>>>>>>>>>>> https://issues.apache.org/jira/issues/?filter=-1&jql=project%3DBEAM%20and%20fixVersion%3D2.38.0%20%20AND%20statusCategory%20!%3D%20Done%20order%20by%20updated%20DESC >>>>>>>>>>>>>> >>>>>>>> [3] >>>>>>>>>>>>>> https://beam.apache.org/contribute/release-blocking/ >>>>>>>>>>>>>> >>>>>>>>>>>>>
