(Ack...resending) Ok, I have the CL ready, if anyone with Java readability would be willing to do a review, please let me know. more inline...
On Fri, Apr 10, 2009 at 2:07 PM, Pam Greene <[email protected]> wrote: > > > On Thu, Apr 9, 2009 at 6:49 PM, Ojan Vafai <[email protected]> wrote: > >> At a quick glance, this looks great. I didn't look over every bug, but the >> ones I did look at look good. > > > Yep. You'll want some sort of default description for the ones that have > none. > I ran the script on a small test file...an example bug is here: http://code.google.com/p/chromium/issues/detail?id=9991 There's at least a small bit of indication of where it came from. I also added the line numbers from test_expectations.txt that generated the bug. Additionally, I also wrote the script to then add the created bug numbers to the file. This ends up re-arranging some of the flags (DEBUG DEFER ... etc. --> DEFER DEBUG ...), but all data is retained. > 200+ bugs is certainly too many, but that's no reason not to file them. > (Sorry, couldn't resist. Seriously, yes, definitely file them. Better in the > issue tracker than getting lost.) > There are probably some improvements to be had to better group some of the bugs, but it's probably not an order of magnitude improvement. I just didn't want to create tons of bugs that were not useful. > > >> It would be great to check in a version of this script that we could run >> when a number of tests fail (e.g. when doing a bad webkit merge). That way, >> we can add them all to the local test_expectations.txt file and have it spit >> out the new results. >> > > Fixing the file we have and moving forward are slightly different use > cases, but yes. In the long run, we shouldn't need any script to fix an > existing bug-less expectations file, only the part that adds bugs for newly > added tests. I'm not sure whether that part should be controlled by adding > the tests to the file and having the script file bugs, or making it a fully > interactive "app": give it a list of files and a description, and it both > changes the file and creates a bug. > There may be a short-term solution and a long-term solution. The short term solution seems to be for someone (assumedly the merger) to add the failing tests to the file, run the script, then commit the file. In the long term, this could be one step, or perhaps be driven right from a "roll DEPS to new version of WebKit" script...right? > >> Really, it would be great if the script filed bugs and then just modified >> test_expectations.txt directly (without committing it). Also, the script >> should remove any comments it moves into bug descriptions. We should get to >> a point where all the comments about layout tests are in the bug >> descriptions themselves instead of in this file. >> > > I disagree with this last part. I'd prefer a brief description to remain in > the file, with any details in the bug. Certainly that's needed for WONTFIX > bugs, where we may not have a bug since there's no work to be tracked, but > it's helpful for others too. I've found it frustrating and time-consuming to > track down when I see big blocks of failures with no explanations at all. > Think of it like the svn checkin comment: enough to have an idea what's > going on right there where you need it, with more detail in the bug for when > you're really digging. > Yep, I modified the script so that it will change test_expectations.txt to have the bug number (you'd just need to commit it afterwards). It's pretty easy to for the script to clip out the comments, but I'm reluctant to, because the script may be over/under zealous in what comments it associates with a layout test. Maybe I can add that behavior as another command-line flag. > > - Pam > > >> I think it would be good to get the script checked in first and then run >> it on the existing test_expectations.txt file. >> >> Ojan >> >> >> On Thu, Apr 9, 2009 at 6:42 PM, Glenn Wilson <[email protected]> wrote: >> >>> Hi Pam & Ojan, >>> I wrote a script that would extract all of the layout tests from >>> test_expectations.txt that we haven't marked as WONTFIX and don't have a bug >>> number. I also tried a simple heuristic to get the context of the layout >>> test via nearby comments....it's not perfect, and I'll have to change some >>> of them by hand, but many of the merge comments are getting picked up. >>> >>> I've also hooked up our library for creating demetrius bugs, so getting >>> bugs made for these should be a matter of running the script with different >>> arguments (I hope). >>> >>> What are your thoughts? Is these as descriptive/accurate as we need? Is >>> 200+ bugs too many? >>> >>> Thanks! >>> Glenn >>> >>> >>> >> --~--~---------~--~----~------------~-------~--~----~ Chromium Developers mailing list: [email protected] View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~----------~----~----~----~------~----~------~--~---
