On Mon, Sep 18, 2017, at 04:58 PM, Clark Boylan wrote: > On Mon, Sep 18, 2017, at 06:43 AM, Andreas Jaeger wrote: > > Just a friendly reminder that the upgrade will happen TODAY, Monday > > 18th, starting at 15:00 UTC. The infra team expects that it takes 8 > > hours, so until 2359 UTC. > > This work was functionally completed at 23:43 UTC. We are now running > Gerrit 2.13.9. There are some cleanup steps that need to be performed in > Infra land, mostly to get puppet running properly again. > > You will also notice that newer Gerrit behaves in some new and exciting > ways. Most of these should be improvements like not needing to reapprove > changes that already have a +1 Workflow but also have a +1 Verified; > recheck should now work for these cases. If you find a new behavior that > looks like a bug please let us know, but we should also work to file > them upstream so that newer Gerrit can address them. > > Feel free to ask us questions if anything else comes up. > > Thank you to everyone that helped with the upgrade. Seems like these get > more and more difficult with each Gerrit release so all the help is > greatly appreciated.
As a followup we have been tracking new fun issues/behaviors in Gerrit and fixing them over the last couple days. Here is an update on where we are currently at. Gerrit emails are slow. You may have noticed that you aren't getting quite as much Gerrit email as before. This is because Gerrit is only sending about one email a minute. Upstream bug is at https://bugs.chromium.org/p/gerrit/issues/detail?id=7261 and we have just got https://review.openstack.org/#/c/505677 merged based on the info in that upstream bug. This won't be applied until we get puppet running on review.openstack.org again (more on that later) and will require another Gerrit service restart. The Gerrit web UI's file editor behaves oddly resulting in what appear to be API timeouts. This also seems to affect gertty. I don't think anyone has dug in far enough to understand what is going on yet. Now for known issues that should be fixed. The Gerrit dashboard creator was using queries that didn't work with new Gerrit query behavior. Sdague got this sorted out quick. The Gerrit event stream changed its ref-updated data and now includes refs/heads/$branchname instead of just $branchname under refName when changes merge. This confused Zuul and meant no post jobs were running. Zuul has been updated to handle this new behavior and post jobs are running. There were no gitweb links. This wasn't caught in testing because we used a test cgit setup on review-dev. Fix here was just to switch to using cgit on review.openstack.org (though the link is still called "gitweb" in the Gerrit UI for reasons). Memory consumption has gone up which initially led to frequent garbage collection which led to 500 errors. We bumped heap memory available to Gerrit up to 48GB (from 30GB) and that seems to have stabilized things. Thankfully while needing more memory it doesn't seem to continuously grow like it did on the old version (which forced us to do semi frequent service restarts). We will have to monitor Gerrit to ensure it is properly stable over time. We could not create new projects in Gerrit. This is because Gerrit 2.12 dropped the --name argument from the create-project command which Gerritlib was using. We have updated Gerritlib to check the Gerrit version and pass the correct arguments to create-project. Unfortunately, we still can't create new projects just yet, this is related to puppet not running on review.openstack.org right now. The gerrit server itself is fine and would puppet except that we force puppet to run on our git mirror farm first to ensure proper mirroring of repos and those have been failing since the CentOS 7.4 release. Once we've got puppet happy we can get back to creating new projects in Gerrit. All the details can be found at https://etherpad.openstack.org/p/gerrit-2.13-issues. Thank you for your patience, Clark __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev