Now that the commits have been recovered and things are almost back to normal, I think it's time to think about how to prevent this kind of incidents in the future.
Our open commit access policy was partly made possible by the idea that any bad commits can be always rolled back. But where I failed to think through was that the changes to refs aren't by themselves version controlled, and so it is possible to lose commits by incorrect ref manipulation, such as "git push -f", or by deleting a branch. I still feel strongly that we maintain the open commit access policy. This is how we've been operating for the longest time, and it's also because otherwise adding/removing developers to repositories would be prohibitively tedious. So my proposal is to write a little program that uses GitHub events API to keep track of push activities in our repositories. For every update to a ref in the repository, we can record the timestamp, SHA1 before and after, the user ID. We can maintain a text file for every ref in every repository, and the program can append lines to it. In other words, effectively recreate server-side reflog outside GitHub. The program should also fetch commits, so that it has a local copy for every commit that ever landed on our repositories. Doing this also allows the program to detect non fast-forward. It should warn us in that situation, plus it will create a ref on the commit locally to prevent it from getting lost. We can then make these repositories accessible via rsync to encourage people to mirror them for backup, or we can make them publicly accessible by hosting them on GitHub as well, although the latter could be confusing. WIth a scheme like this, pushes can be safely recorded within a minute or so (and this number can go down even further if we use webhooks.) If a data loss occurs before the program gets to record newly pushed commits, we should still be able to record who pushed afterward to identify who has the commits that were lost. With such a small time window between the push and the record, the number of such lost commits should be low enough such that we can recover them manually. -- Kohsuke Kawaguchi -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
