Re: [chromium-dev] revised output for run_webkit_tests
On Thu, Dec 10, 2009 at 11:28 PM, David Levin le...@chromium.org wrote: On Thu, Dec 10, 2009 at 10:57 PM, Dirk Pranke dpra...@chromium.org wrote: We could do this, but we'd have to add logic to track when directories were done, and arbitrarily delay printing results about other directories (hence delaying and serializing results). This might end up causing weirdly irregular bursts of output. The irregular bursts of output isn't that bad. (I had a version of run-webkit-test that did this. Unfortunately, perl is not a fun language for me at least, and I have to admit that the perl code I had would have been hard to maintain/fragile.) Worst case, since we intentionally run the http tests first, and they're the long pole in the run, this might delay printing all the other directories until near the end. Not a big deal either. My version did this as well. (I started this behavior in my webkit version and talked to Ojan about doing it.) Well, I can certainly try to hack up a version of the script that generates the output you're looking for to see how it works. I'm not sure what the real benefit of this would be. The benefit is working in a community and understanding how they do things and adapting to that as opposed to trying to push something very different on them. Sure. Of course, there's not necessarily a reason to leave things the same just because that's the way it's always been done, especially when trying to keep things the same imposes costs. Sometimes different is better. (A) Have you looked at the new output yet? (B) Is getting output by directory really that useful? I understood your description. Having run the webkit version, I much prefer it due to knowing when certain directories are done and knowing what test(s) failed in those directories as the test goes along (even in the parallel version where the failures may be slightly delayed). The output by directory also adapts better to the buildbot output instead of the huge test-by-test list that chromium buildbots have (which takes a while to download when you click the link for stdio). There's no arguing that the --verbose output is in fact extremely verbose. But, that is because there is a lot of information there. Personally, I find the webkit output a bad compromise between terseness and verbosity - it's too much text for interactive use where you expect things to pass, and too little if you actually want to debug what happened. In particular I think this would be very true in a multithreaded scenarios, since you would lose any grasp of what was actually happening in parallel. The current implementation tells you that tests have failed as it goes, but not which tests (of course, the webkit script doesn't tell you either, apart from which directory the failure might be in). That would be easy to add if there is demand. dave -- Dirk On Thu, Dec 10, 2009 at 10:10 PM, David Levin le...@chromium.org wrote: Actually, you can have a similar output even with the multi-threading. You can display the results for one only directory (even though multiple directories are being processed at the same time) and when that directory is done, display the results for the next directory until it is done, etc. The ordering of the directories may be different but the output is very similar to what they have now. The effect is quite satisfying and clear. dave On Thu, Dec 10, 2009 at 10:04 PM, Dirk Pranke dpra...@chromium.org wrote: Yes, I did consider that. The fatal flaw in that plan is that the webkit test script is single-threaded and runs through the tests in order. Ours doesn't, and so we can't easily guarantee the same sort of output they have. Eric and I will probably work through this as we upstream the code. I'm actually hoping to get them to adopt my output, but we'll see. -- Dirk On Thu, Dec 10, 2009 at 7:45 PM, David Levin le...@chromium.org wrote: Have you considered making the output closer to that of WebKit's run-webkit-tests? It seems that would ease the hopeful transition to this version upstream. dave On Thu, Dec 10, 2009 at 7:23 PM, Dirk Pranke dpra...@chromium.org wrote: Hi all, If you never run the webkit layout tests, you can stop reading. Otherwise, earlier today I checked in a patch that should make the output much less verbose in the normal case. From the CL: First, a number of log messages have had their levels changed (mostly to make them quieter). Second, the script outputs a meter that shows progress through the test run, which is a one line summary of where it's at current (e.g. parsing expectations, gathering files. During the actual test execution, the meter displays %d tests completed as expected, %d didn't, %d remain. The meter uses carriage returns but no linefeeds, so the output is overwritten as it progresses. The meter is disabled if
Re: [chromium-dev] revised output for run_webkit_tests
Sigh. Now from the right email address. On Fri, Dec 11, 2009 at 11:36 AM, Ojan Vafai o...@google.com wrote: I thought we had agreed on printing out any unexpected failures in real-time, no? Also, I do think it would be worthwhile to print each directory as it finishes. We're getting to the point where we shard all the big directories, so the largest shard takes 90 seconds (this is true on the mac release bot now!). So printing directories as they finish would actually give you decent insight into what tests have been run. Ojan On Fri, Dec 11, 2009 at 9:55 AM, Dirk Pranke dpra...@chromium.org wrote: On Fri, Dec 11, 2009 at 12:04 AM, Dirk Pranke dpra...@chromium.org wrote: The current implementation tells you that tests have failed as it goes, but not which tests (of course, the webkit script doesn't tell you either, apart from which directory the failure might be in). That would be easy to add if there is demand. It has been pointed out to me that this paragraph is incorrect. The webkit scripts do give you the test filename of the failure in real time. For some reason I was thinking it just printed an E instead of a . . Thinking about it now, I think I got its output confused with the Python and Perl unit testing frameworks ... My apologies for any confusion this might've caused, and for saying something without being sure it was right ... -- Dirk -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
Re: [chromium-dev] revised output for run_webkit_tests
On Fri, Dec 11, 2009 at 11:36 AM, Ojan Vafai o...@google.com wrote: I thought we had agreed on printing out any unexpected failures in real-time, no? Also, I do think it would be worthwhile to print each directory as it finishes. We're getting to the point where we shard all the big directories, so the largest shard takes 90 seconds (this is true on the mac release bot now!). So printing directories as they finish would actually give you decent insight into what tests have been run. No, I don't think we had necessarily agreed to this, although we did talk about it. At any rate, I'm working on a webkit-style-output patch that should get both of these things, and we can see which we like better. -- Dirk -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
Re: [chromium-dev] revised output for run_webkit_tests
Have you considered making the output closer to that of WebKit's run-webkit-tests? It seems that would ease the hopeful transition to this version upstream. dave On Thu, Dec 10, 2009 at 7:23 PM, Dirk Pranke dpra...@chromium.org wrote: Hi all, If you never run the webkit layout tests, you can stop reading. Otherwise, earlier today I checked in a patch that should make the output much less verbose in the normal case. From the CL: First, a number of log messages have had their levels changed (mostly to make them quieter). Second, the script outputs a meter that shows progress through the test run, which is a one line summary of where it's at current (e.g. parsing expectations, gathering files. During the actual test execution, the meter displays %d tests completed as expected, %d didn't, %d remain. The meter uses carriage returns but no linefeeds, so the output is overwritten as it progresses. The meter is disabled if --verbose is specified, to avoid unnecessary confusion. Third, I removed the --find-baselines option. I think I was the only one using it, and --sources is good enough (but added the baseline for the checksum as well as the .png when using --sources). Fourth, there is a new --log option that can be used to provide finer granularity of logging. It accepts a comma-separated list of options, like: --log 'actual,expected,timing': actual: the actual test results (# of failures by type and timeline) config: the test settings (results dir, platform, etc.) expected: the results we expected by type and timeline timing: test timing results (slow files, total execution, etc.) All of this information is logged at the logging.info level (if the appropriate option is enabled). Using the --verbose switch will cause all of options to be logged, as well as the normal verbose output. In addition, the verbose output will disable the meter (as mentioned above). Note that the actual results will be logged to stdout, not stderr, for compatibility with the buildbot log parser. Finally, the list of unexpected results (if any) will be logged to stdout, along with a one-line summary of the test run. The net result is that when run with no command line options (and when no tests fail), only one line of output will be produced. Feedback / problems / questions to me. Pam, sorry for making all of your examples in your tech talk immediately out of date :) -- Dirk -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
Re: [chromium-dev] revised output for run_webkit_tests
Yes, I did consider that. The fatal flaw in that plan is that the webkit test script is single-threaded and runs through the tests in order. Ours doesn't, and so we can't easily guarantee the same sort of output they have. Eric and I will probably work through this as we upstream the code. I'm actually hoping to get them to adopt my output, but we'll see. -- Dirk On Thu, Dec 10, 2009 at 7:45 PM, David Levin le...@chromium.org wrote: Have you considered making the output closer to that of WebKit's run-webkit-tests? It seems that would ease the hopeful transition to this version upstream. dave On Thu, Dec 10, 2009 at 7:23 PM, Dirk Pranke dpra...@chromium.org wrote: Hi all, If you never run the webkit layout tests, you can stop reading. Otherwise, earlier today I checked in a patch that should make the output much less verbose in the normal case. From the CL: First, a number of log messages have had their levels changed (mostly to make them quieter). Second, the script outputs a meter that shows progress through the test run, which is a one line summary of where it's at current (e.g. parsing expectations, gathering files. During the actual test execution, the meter displays %d tests completed as expected, %d didn't, %d remain. The meter uses carriage returns but no linefeeds, so the output is overwritten as it progresses. The meter is disabled if --verbose is specified, to avoid unnecessary confusion. Third, I removed the --find-baselines option. I think I was the only one using it, and --sources is good enough (but added the baseline for the checksum as well as the .png when using --sources). Fourth, there is a new --log option that can be used to provide finer granularity of logging. It accepts a comma-separated list of options, like: --log 'actual,expected,timing': actual: the actual test results (# of failures by type and timeline) config: the test settings (results dir, platform, etc.) expected: the results we expected by type and timeline timing: test timing results (slow files, total execution, etc.) All of this information is logged at the logging.info level (if the appropriate option is enabled). Using the --verbose switch will cause all of options to be logged, as well as the normal verbose output. In addition, the verbose output will disable the meter (as mentioned above). Note that the actual results will be logged to stdout, not stderr, for compatibility with the buildbot log parser. Finally, the list of unexpected results (if any) will be logged to stdout, along with a one-line summary of the test run. The net result is that when run with no command line options (and when no tests fail), only one line of output will be produced. Feedback / problems / questions to me. Pam, sorry for making all of your examples in your tech talk immediately out of date :) -- Dirk -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
Re: [chromium-dev] revised output for run_webkit_tests
On Thu, Dec 10, 2009 at 10:57 PM, Dirk Pranke dpra...@chromium.org wrote: We could do this, but we'd have to add logic to track when directories were done, and arbitrarily delay printing results about other directories (hence delaying and serializing results). This might end up causing weirdly irregular bursts of output. The irregular bursts of output isn't that bad. (I had a version of run-webkit-test that did this. Unfortunately, perl is not a fun language for me at least, and I have to admit that the perl code I had would have been hard to maintain/fragile.) Worst case, since we intentionally run the http tests first, and they're the long pole in the run, this might delay printing all the other directories until near the end. Not a big deal either. My version did this as well. (I started this behavior in my webkit version and talked to Ojan about doing it.) I'm not sure what the real benefit of this would be. The benefit is working in a community and understanding how they do things and adapting to that as opposed to trying to push something very different on them. (A) Have you looked at the new output yet? (B) Is getting output by directory really that useful? I understood your description. Having run the webkit version, I much prefer it due to knowing when certain directories are done and knowing what test(s) failed in those directories as the test goes along (even in the parallel version where the failures may be slightly delayed). The output by directory also adapts better to the buildbot output instead of the huge test-by-test list that chromium buildbots have (which takes a while to download when you click the link for stdio). dave -- Dirk On Thu, Dec 10, 2009 at 10:10 PM, David Levin le...@chromium.org wrote: Actually, you can have a similar output even with the multi-threading. You can display the results for one only directory (even though multiple directories are being processed at the same time) and when that directory is done, display the results for the next directory until it is done, etc. The ordering of the directories may be different but the output is very similar to what they have now. The effect is quite satisfying and clear. dave On Thu, Dec 10, 2009 at 10:04 PM, Dirk Pranke dpra...@chromium.org wrote: Yes, I did consider that. The fatal flaw in that plan is that the webkit test script is single-threaded and runs through the tests in order. Ours doesn't, and so we can't easily guarantee the same sort of output they have. Eric and I will probably work through this as we upstream the code. I'm actually hoping to get them to adopt my output, but we'll see. -- Dirk On Thu, Dec 10, 2009 at 7:45 PM, David Levin le...@chromium.org wrote: Have you considered making the output closer to that of WebKit's run-webkit-tests? It seems that would ease the hopeful transition to this version upstream. dave On Thu, Dec 10, 2009 at 7:23 PM, Dirk Pranke dpra...@chromium.org wrote: Hi all, If you never run the webkit layout tests, you can stop reading. Otherwise, earlier today I checked in a patch that should make the output much less verbose in the normal case. From the CL: First, a number of log messages have had their levels changed (mostly to make them quieter). Second, the script outputs a meter that shows progress through the test run, which is a one line summary of where it's at current (e.g. parsing expectations, gathering files. During the actual test execution, the meter displays %d tests completed as expected, %d didn't, %d remain. The meter uses carriage returns but no linefeeds, so the output is overwritten as it progresses. The meter is disabled if --verbose is specified, to avoid unnecessary confusion. Third, I removed the --find-baselines option. I think I was the only one using it, and --sources is good enough (but added the baseline for the checksum as well as the .png when using --sources). Fourth, there is a new --log option that can be used to provide finer granularity of logging. It accepts a comma-separated list of options, like: --log 'actual,expected,timing': actual: the actual test results (# of failures by type and timeline) config: the test settings (results dir, platform, etc.) expected: the results we expected by type and timeline timing: test timing results (slow files, total execution, etc.) All of this information is logged at the logging.info level (if the appropriate option is enabled). Using the --verbose switch will cause all of options to be logged, as well as the normal verbose output. In addition, the verbose output will disable the meter (as mentioned above). Note that the actual results will be logged to stdout, not stderr, for compatibility with the buildbot log parser. Finally, the list of unexpected
