On 06/03/2018 19:44, Marko Obrovac wrote:
> Hello,
> Today during the scheduled deploy window of moving JobQueue jobs from
> the Redis-based infrastructure to the new EventBus-based one, a
> combination of PEBKAC and Gerrit trying to be smart caused an accidental
> merge of the mw core's master branch into wmf.23. Luckily, the mistake
> was spotted before the wmf.23 branch got updated on tin, so the blast
> radius was limited to delaying the Euro-SWAT window by 30 minutes~[1].
> Big thanks to Antoine and Giuseppe for their help in discovering and
> resolving the issues.
> You can read the full incident report here~[2]. The TL;DR is that
> currently MW core repository is configured in such a way so as to allow
> Gerrit to do implicit (and silent!) merges, even for cherry-picked
> change-sets, which is something we might want to change in the near future.
> Cheers,
> Marko
> [1] Thanks, James F, for patiently waiting on the resolution!
> [2] 
> https://wikitech.wikimedia.org/wiki/Incident_documentation/20180306-MasterMergedInWMFBranch
> Marko Obrovac, PhD
> Senior Services Engineer
> Wikimedia Foundation


That Gerrit feature has been disabled by default for all repositories by
setting receive.rejectImplicitMerges = true in All-Projects.


Implicit merging of a branch into another might be helpful sometime, but
most probably all typical use cases would be due to a mistake and could
lead to a catastrophe.  I blame Gerrit default behavior on that one :]

I would like to thanks Marko and Petr to have followed the deployment guide:
* always check current workspace and remote before rebasing:
   git remote update
   git log HEAD..HEAD@{u}
* halt immediately and ring the bell in case of doubt. Surely have ~90
unwanted commits was a red alarm.

Thank you!

Antoine Musso

Services mailing list

Reply via email to