- If you have issues with gcl or gclient misbehaving, reads the "End Results" section.- If you don't care about the build breakage and code review tool screw up yesterday, you can stop reading now. - If you want to laugh about me, continue reading.
Timeline (all times in PDT) *On Fri Apr 17 at 6:49, r13929<http://src.chromium.org/viewvc/chrome?view=rev&revision=13929> * adds git-cl and git-try to depot_tools. Consequence: They're broken so for git users that have depot_tools in the path before the actual git-cl and git-try. Error #1: This change should have been using a canary user or at least barely tested first. *On Tue Ap 21 at 6:51, r14094<http://src.chromium.org/viewvc/chrome?view=rev&revision=14094> * adds default presubmit.py checks to gcl.py. Consequence: This is breaking gcl.py because it looks for PRESUBMIT.py in the repository root and caches it in .svn/gcl_info/. The failure is because once this file is cached, *on the next run*, gcl.py loads this cached file as a *change list* and spits out an exception in that case.Consequence: This also effectively starts to totally ignore codereview.settings on the svn repo because of a bug in GetCachedFile(), causing all weird issues because it totally ignores the repo settings. This includes ignoring the "tree is closed" flag. *At 8:33, r14101<http://src.chromium.org/viewvc/chrome?view=rev&revision=14101> * upgrades the windows svn client included in depot_tools to 1.6.1 from 1.5.4. This changes can't be easily reverted since once svn 1.6 runs on a checkout, it changes the working format. This is mainly to help TortoiseSVN users that upgraded to svn 1.6. Consequence: This is breaking both gclient.py and gcl.py because the output of `svn status` was changed, introducing a whitespace in file names and confusing both scripts. This caused a total try server failure but didn't adversely affect the continuous build. Consequence: This causes a non-fatal build error in chrome_dll.vcproj because the svn client included in third_party/svn is still version 1.5.0 (there shouldn't be a svn client in chromium sources as a starter but that's another story) Consequence: Nobody can upload a change list or commit it. Error #2: This should have been done over the week end as usual, reducing the damages. Error #3: I should have acted as a canary myself a bit so I would have found about the whitespace issue and the next one. Error #4: All Windows client users were forced to move to svn 1.6 checkout format without prior notice. *At 10:07*, I receive the first notifications about some weirdness in gcl behavior. After talking with the user, it was deemed that he was using an older version and just needed updating to r14094; which was false, it was * caused* by r14094. *At 10:47, r14120<http://src.chromium.org/viewvc/chrome?view=rev&revision=14120> * tries to fix the whitespace issue in both gcl.py and gclient.py. Consequence: This caused a total continuous build and try server failure which was just worse than before because of an exception in gclient.py. An exception in gclient.py means deleting the whole checkout on the next build run as a safely measure. The total number of try jobs lost wasn't that high but the continuous build down time was really atrocious; some slave taking more than 1 hour to simply do the checkout (update) step. Then add a full rebuild for every builders. This slowness has been caused by an independent network configuration issue that should be fixed soon. Error #5: Simply took too much time to figure out what the problem was, e.g. I was gone lunching. Error #6: The fix wasn't tested enough and just broke the tree even more. *At 10:55, r14122<http://src.chromium.org/viewvc/chrome?view=rev&revision=14122> * fixed gclient.py exception. I could start breathing again. Well, almost. *At 12:35, r14126<http://src.chromium.org/viewvc/chrome?view=rev&revision=14126> * fixes the issue introduced in r14094. This fixes only the GetCachedFile() caching error introduced in r14094, so that codereview.settings (and potentially PRESUBMIT.py if it had existed) can now be cached correctly. *At 12:42*, I send an email to various people to warn them to delete their .svn/gcl_info/codereview.settings file. Consequence: People can't still use gcl on the second run because of the .svn/gcl_info/PRESUBMIT.py file that keeps on being created. *At 16:58*, after discussing with me,* thestig@ commits r14160<http://src.chromium.org/viewvc/chrome?view=rev&revision=14160> * to revert both r14094 and r14126. That effectively fixes the remaining issue that .svn/gcl_info/PRESUBMIT.py is created but that doesn't remove the file for user affected by this issue. *At 16:39*, evan@ sends an email about the PRESUBMIT.py issue and notes git-cl and git-try are deeply broken. (I only notice the day after) *On Wed Apr 22 at 7:24, r14186<http://src.chromium.org/viewvc/chrome?view=rev&revision=14186> * fix the git-cl and git-try breakage introduced in r13929. End Results - There is still a conflicting svn version in third_party/svn. - Some user are still affected by having bad files living in .svn/gcl_info/. They need to *manually* remove them. - A lot of unnecessary down time. Improvements - Give prior notice for check-ins that are known to be potentially failing. - Check-in more often the weekend the depot_tools. - Add a unit test to gcl. - Use `svn status --xml` which is know to not change between revisions. M-A --~--~---------~--~----~------------~-------~--~----~ Chromium Developers mailing list: [email protected] View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~----------~----~----~----~------~----~------~--~---
