On Wed, 29 Jun 2022 at 22:03, Michael Bien <mbie...@gmail.com> wrote: > I believe part of the problem was that we had issues with 2 of 3 build > clusters. Travis didn't want to build [1] (basically just like right now > when we have a queue time of ~24h for some reason) and Jenkins had > problems too which is probably even worse during release. > > This caused PRs to pile up and added at least one extra RC, which in > turn also added more opportunities to integrate more PRs etc.
Yes, didn't mention Jenkins infrastructure issues. Eric, do you know whether what happened is fully resolved / could happen again? I'm not sure the infrastructure issues were the reason for RC5 (never actually announced), but did allow a couple more things in for RC6 while it was being resolved. > There was also the incident when we lost ~2 PRs from the delivery branch > which also caused an extra RC. To avoid this in future, we should double > check the sync PRs before merging, to make sure everything is in it. > Lets do a proper review as if it is a regular PR (maybe put the authors > of the PRs which got merged during RC on the reviewer list?). I'm not sure I'd add that overhead into turning around the sync PRs in the general case, given that sometimes we need to turn around in a few hours. We've not had this issue before, and I think the reason was clear. We've never had to force push delivery before, although it's always been an outside possibility. I think we had 2 this time, the first of which possibly shouldn't have been necessary and a sign there was a problem. I'm not sure we can lose a PR without a force push? I think we need more robust checks and [NOTICE] emails if there's a need to force push, so full review in those circumstances would make sense. Of course, PR authors should probably also be checking on sync to release and master that changes merged, applied and work correctly anyway! We could potentially use GitHub actions to automatically notify on any force push into a repository branch. That might be worth considering? > We should also spend more time trying to make tests fast and reliable > since this is actually something we can influence. Long queue times > become less of a problem if we don't have to restart jobs like playing > whack a mole ;) You take away the need to whack a mole on Travis and you take away half the "fun" of managing releases! :-) Best wishes, Neil --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@netbeans.apache.org For additional commands, e-mail: dev-h...@netbeans.apache.org For further information about the NetBeans mailing lists, visit: https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
