On Thu, Dec 10, 2009 at 11:28 PM, David Levin <[email protected]> wrote: > > > On Thu, Dec 10, 2009 at 10:57 PM, Dirk Pranke <[email protected]> wrote: >> >> We could do this, but we'd have to add logic to track when directories >> were "done", and arbitrarily delay printing results about "other" >> directories (hence delaying and serializing results). This might end >> >> up causing weirdly irregular bursts of output. > > The irregular bursts of output isn't that bad. (I had a version of > run-webkit-test that did this. Unfortunately, perl is not a fun language > for me at least, and I have to admit that the perl code I had would have > been hard to maintain/fragile.) >> >> Worst case, since we >> intentionally run the "http" tests first, and they're the long pole in >> the run, this might delay printing all the other directories until >> near the end. > > Not a big deal either. My version did this as well. (I started this behavior > in my webkit version and talked to Ojan about doing it.) >
Well, I can certainly try to hack up a version of the script that generates the output you're looking for to see how it works. >> I'm not sure what the real benefit of this would be. > > > The benefit is working in a community and understanding how they do things > and adapting to that as opposed to trying to push something very different > on them. Sure. Of course, there's not necessarily a reason to leave things the same just because "that's the way it's always been done", especially when trying to keep things the same imposes costs. Sometimes different is better. >> >> (A) Have you looked at the new output yet? >> >> (B) Is getting output by directory really that useful? > > I understood your description. Having run the webkit version, I much prefer > it due to knowing when certain directories are done and knowing what test(s) > failed in those directories as the test goes along (even in the parallel > version where the failures may be slightly delayed). > The output by directory also adapts better to the buildbot output instead of > the huge test-by-test list that chromium buildbots have (which takes a while > to download when you click the link for stdio). There's no arguing that the --verbose output is in fact extremely verbose. But, that is because there is a lot of information there. Personally, I find the webkit output a bad compromise between terseness and verbosity - it's too much text for interactive use where you expect things to pass, and too little if you actually want to debug what happened. In particular I think this would be very true in a multithreaded scenarios, since you would lose any grasp of what was actually happening in parallel. The current implementation tells you that tests have failed as it goes, but not which tests (of course, the webkit script doesn't tell you either, apart from which directory the failure might be in). That would be easy to add if there is demand. > dave >> >> -- Dirk >> >> On Thu, Dec 10, 2009 at 10:10 PM, David Levin <[email protected]> wrote: >> > Actually, you can have a similar output even with the multi-threading. >> > You can display the results for one only directory (even though multiple >> > directories are being processed at the same time) and when that >> > directory is >> > done, display the results for the next directory until it is done, >> > etc. The >> > ordering of the directories may be different but the output is very >> > similar >> > to what they have now. >> > The effect is quite satisfying and clear. >> > dave >> > On Thu, Dec 10, 2009 at 10:04 PM, Dirk Pranke <[email protected]> >> > wrote: >> >> >> >> Yes, I did consider that. The fatal flaw in that plan is that the >> >> webkit test script is single-threaded and runs through the tests in >> >> order. Ours doesn't, and so we can't easily guarantee the same sort of >> >> output they have. Eric and I will probably work through this as we >> >> upstream the code. I'm actually hoping to get them to adopt my output, >> >> but we'll see. >> >> >> >> -- Dirk >> >> >> >> On Thu, Dec 10, 2009 at 7:45 PM, David Levin <[email protected]> >> >> wrote: >> >> > Have you considered making the output closer to that of WebKit's >> >> > run-webkit-tests? >> >> > It seems that would ease the hopeful transition to this version >> >> > upstream. >> >> > dave >> >> > On Thu, Dec 10, 2009 at 7:23 PM, Dirk Pranke <[email protected]> >> >> > wrote: >> >> >> >> >> >> Hi all, >> >> >> >> >> >> If you never run the webkit layout tests, you can stop reading. >> >> >> >> >> >> Otherwise, earlier today I checked in a patch that should make the >> >> >> output much less verbose in the normal case. From the CL: >> >> >> >> >> >> First, a number of log messages have had their levels changed >> >> >> (mostly >> >> >> to >> >> >> make them quieter). >> >> >> >> >> >> Second, the script outputs a "meter" that shows progress through the >> >> >> test run, which is a one line summary of where it's at current >> >> >> (e.g. "parsing expectations", "gathering files". During the actual >> >> >> test >> >> >> execution, the meter displays "%d tests completed as expected, %d >> >> >> didn't, >> >> >> %d remain". The meter uses carriage returns but no linefeeds, so the >> >> >> output >> >> >> is overwritten as it progresses. The meter is disabled if --verbose >> >> >> is >> >> >> specified, to avoid unnecessary confusion. >> >> >> >> >> >> Third, I removed the --find-baselines option. I think I was the only >> >> >> one >> >> >> using it, and --sources is good enough (but added the baseline for >> >> >> the checksum as well as the .png when using --sources). >> >> >> >> >> >> Fourth, there is a new "--log" option that can be used to provide >> >> >> finer >> >> >> granularity of logging. It accepts a comma-separated list of >> >> >> options, >> >> >> like: >> >> >> --log 'actual,expected,timing': >> >> >> >> >> >> "actual": the actual test results (# of failures by type and >> >> >> timeline) >> >> >> "config": the test settings (results dir, platform, etc.) >> >> >> "expected": the results we expected by type and timeline >> >> >> "timing": test timing results (slow files, total execution, etc.) >> >> >> >> >> >> All of this information is logged at the logging.info level (if the >> >> >> appropriate option is enabled). >> >> >> >> >> >> Using the --verbose switch will cause all of options to be logged, >> >> >> as >> >> >> well >> >> >> as the normal verbose output. In addition, the verbose output will >> >> >> disable >> >> >> the meter (as mentioned above). Note that the "actual" results will >> >> >> be >> >> >> logged >> >> >> to stdout, not stderr, for compatibility with the buildbot log >> >> >> parser. >> >> >> >> >> >> Finally, the list of unexpected results (if any) will be logged to >> >> >> stdout, >> >> >> along with a one-line summary of the test run. >> >> >> >> >> >> The net result is that when run with no command line options (and >> >> >> when >> >> >> no >> >> >> tests fail), only one line of output will be produced. >> >> >> >> >> >> Feedback / problems / questions to me. >> >> >> >> >> >> Pam, sorry for making all of your examples in your tech talk >> >> >> immediately out of date :) >> >> >> >> >> >> -- Dirk >> >> >> >> >> >> -- >> >> >> Chromium Developers mailing list: [email protected] >> >> >> View archives, change email options, or unsubscribe: >> >> >> http://groups.google.com/group/chromium-dev >> >> > >> >> > >> > >> > > > -- Chromium Developers mailing list: [email protected] View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
