[webkit-dev] Pixel tests and displaying text
If you never write layout tests, you can stop reading. Right now few ports run pixel tests (a shame). One reason is because we frequently need different reference images for each port, and creating hundreds or thousands of these is a hassle. Maintenance burden also increases. We could greatly decrease the number of these baselines by following a simple rule: don't display text unless you need to. For example, let's say I have a test that is supposed to display a green square. If the test output is: A green square means this test passes. If there is any red below, fail. [green square here] ...then for each platform that has different font rendering, whether that be subpixel AA differences, font size differences, or anything else, we'll need new baselines. By contrast, if that explanation was in an HTML comment inside the test code, and the output was a simple green square, one baseline would serve for all ports. I think this doesn't really hinder tracking down test failures, for a couple of reasons: * Most tests are green = pass, red = failure by convention, so frequently a pixel result that differs from the baseline is easily classifiable at a quick glance. * When this isn't the case, one of the first steps in understanding the test output is to read the test, at which point the HTML comment will explain things. Obviously, some tests need to display text, but this seems to me to be a good rule of thumb. Am I missing anything? If not, is anyone interested in helping to make this change on existing tests where possible, to trim the number of baselines in the tree and make it easier for new ports to bring up pixel tests? PK ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Pixel tests and displaying text
I think we should just switch the default for text vs. pixel tests. Make tests require a script layoutTestController.dumpPixels()/script to dump pixels. We could also white-list certain directories.[1] -eric 1. If we did add white-listing, I'd like to move away from our hacky-in-run-webkit-tests white-lists to an explict on-disk per-directory config file or similar. Understanding why fast/loader turns on the loader callbacks when you're writing a test has driven more than one would-be-webkit-hacker crazy. :) On Wed, Dec 8, 2010 at 12:43 PM, Simon Fraser simon.fra...@apple.comwrote: On Dec 8, 2010, at 12:13 PM, Peter Kasting wrote: If you never write layout tests, you can stop reading. Right now few ports run pixel tests (a shame). One reason is because we frequently need different reference images for each port, and creating hundreds or thousands of these is a hassle. Maintenance burden also increases. We could greatly decrease the number of these baselines by following a simple rule: don't display text unless you need to. For example, let's say I have a test that is supposed to display a green square. If the test output is: A green square means this test passes. If there is any red below, fail. [green square here] ...then for each platform that has different font rendering, whether that be subpixel AA differences, font size differences, or anything else, we'll need new baselines. By contrast, if that explanation was in an HTML comment inside the test code, and the output was a simple green square, one baseline would serve for all ports. I think this doesn't really hinder tracking down test failures, for a couple of reasons: * Most tests are green = pass, red = failure by convention, so frequently a pixel result that differs from the baseline is easily classifiable at a quick glance. * When this isn't the case, one of the first steps in understanding the test output is to read the test, at which point the HTML comment will explain things. Obviously, some tests need to display text, but this seems to me to be a good rule of thumb. Am I missing anything? If not, is anyone interested in helping to make this change on existing tests where possible, to trim the number of baselines in the tree and make it easier for new ports to bring up pixel tests? I totally agree, and have been avoiding test in pixel tests as much as possible recently. One useful strategy is to add explanatory notes to the test as HTML comments, so the purpose of the test is easily determined by someone looking at the source. Simon ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Pixel tests and displaying text
On Dec 8, 2010, at 12:13 PM, Peter Kasting wrote: If you never write layout tests, you can stop reading. Right now few ports run pixel tests (a shame). One reason is because we frequently need different reference images for each port, and creating hundreds or thousands of these is a hassle. Maintenance burden also increases. We could greatly decrease the number of these baselines by following a simple rule: don't display text unless you need to. For example, let's say I have a test that is supposed to display a green square. If the test output is: A green square means this test passes. If there is any red below, fail. [green square here] ...then for each platform that has different font rendering, whether that be subpixel AA differences, font size differences, or anything else, we'll need new baselines. By contrast, if that explanation was in an HTML comment inside the test code, and the output was a simple green square, one baseline would serve for all ports. I think this doesn't really hinder tracking down test failures, for a couple of reasons: * Most tests are green = pass, red = failure by convention, so frequently a pixel result that differs from the baseline is easily classifiable at a quick glance. * When this isn't the case, one of the first steps in understanding the test output is to read the test, at which point the HTML comment will explain things. Obviously, some tests need to display text, but this seems to me to be a good rule of thumb. Tests that involve text can still have cross-platform results if they use the Ahem font. See for example http://trac.webkit.org/browser/trunk/LayoutTests/fast/css/first-letter-set-text.html?rev=55196. Am I missing anything? If not, is anyone interested in helping to make this change on existing tests where possible, to trim the number of baselines in the tree and make it easier for new ports to bring up pixel tests? PK ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Pixel tests and displaying text
On Dec 8, 2010, at 12:13 PM, Peter Kasting wrote: We could greatly decrease the number of these baselines by following a simple rule: don't display text unless you need to. Seems like a good idea. I’m worried a bit, though, that if we can’t use any text in them at all, the tests are then not at all self explanatory. You have to be an expert on the test to understand what it’s testing and what success and failure look like. Maybe we can come up with a new form of test that puts the explanation of the test somewhere that will not be dumped and the test in a frame, and have the pixel tests dump only the frame. -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Pixel tests and displaying text
On Wed, Dec 8, 2010 at 12:57 PM, Darin Adler da...@apple.com wrote: I’m worried a bit, though, that if we can’t use any text in them at all, the tests are then not at all self explanatory. You have to be an expert on the test to understand what it’s testing and what success and failure look like. I think for tests where we're worried that's the case, we should (for now) just go ahead and continue using text and having separate baselines. The per-test cost is relatively low and the number of tests that fall into this bucket is hopefully small? Maybe we can come up with a new form of test that puts the explanation of the test somewhere that will not be dumped and the test in a frame, and have the pixel tests dump only the frame. That would also work. PK ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Pixel tests and displaying text
On Dec 8, 2010, at 1:02 PM, Peter Kasting wrote: the number of tests that fall into this bucket is hopefully small? Maybe we should talk about some specific tests. I can‘t think of many tests with no text that are self explanatory! -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Pixel tests and displaying text
On Wed, Dec 8, 2010 at 1:04 PM, Darin Adler da...@apple.com wrote: On Dec 8, 2010, at 1:02 PM, Peter Kasting wrote: the number of tests that fall into this bucket is hopefully small? Maybe we should talk about some specific tests. I can‘t think of many tests with no text that are self explanatory! It's not so much whether the test is self-explanatory as whether, given a result that differs from the baseline, it's obvious whether it's a failure and how to proceed. The particular change that triggered this thread, for example, was http://trac.webkit.org/changeset/73531 . The tests added in this commit are basically of the standard some number of green squares; red = failure form. Moving the explanatory text from them into the HTML for the tests themselves would not greatly decrease the ease with which someone could understand why they'd suddenly started failing the tests. Text is more useful when it's more likely that differences will occur that are not obviously either OK or wrong. In that case, having the text in the test output saves a trip to the HTML source on the part of the person trying to figure out whether he's seeing a failure. PK ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Pixel tests and displaying text
On Dec 8, 2010, at 12:57 PM, Darin Adler wrote: On Dec 8, 2010, at 12:13 PM, Peter Kasting wrote: We could greatly decrease the number of these baselines by following a simple rule: don't display text unless you need to. Seems like a good idea. I’m worried a bit, though, that if we can’t use any text in them at all, the tests are then not at all self explanatory. You have to be an expert on the test to understand what it’s testing and what success and failure look like. Maybe we can come up with a new form of test that puts the explanation of the test somewhere that will not be dumped and the test in a frame, and have the pixel tests dump only the frame. Do HTML comments fulfill this? Simon ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Pixel tests and displaying text
On Dec 8, 2010, at 1:23 PM, Simon Fraser wrote: On Dec 8, 2010, at 12:57 PM, Darin Adler wrote: On Dec 8, 2010, at 12:13 PM, Peter Kasting wrote: We could greatly decrease the number of these baselines by following a simple rule: don't display text unless you need to. Seems like a good idea. I’m worried a bit, though, that if we can’t use any text in them at all, the tests are then not at all self explanatory. You have to be an expert on the test to understand what it’s testing and what success and failure look like. Do HTML comments fulfill this? Peter, Simon, You both have convinced me that HTML comments could be fine for explaining what a test is testing. Maybe we could come up with a format that makes it likely we’ll spot those comments. The success vs. failure aspects can probably be addressed by the “see no red” and other such things. -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Pixel tests and displaying text
On Wed, Dec 8, 2010 at 2:03 PM, Darin Adler da...@apple.com wrote: You both have convinced me that HTML comments could be fine for explaining what a test is testing. Maybe we could come up with a format that makes it likely we’ll spot those comments. In the (few) tests I've written, I generally use top-of-file comments that describe both the expected results and what the underlying causes for potential failures are. See for example the layout test at the bottom of https://bugs.webkit.org/attachment.cgi?id=72387action=review . PK ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Pixel tests and displaying text
On Dec 8, 2010, at 1:02 PM, Peter Kasting wrote: On Wed, Dec 8, 2010 at 12:57 PM, Darin Adler da...@apple.com wrote: I’m worried a bit, though, that if we can’t use any text in them at all, the tests are then not at all self explanatory. You have to be an expert on the test to understand what it’s testing and what success and failure look like. I think for tests where we're worried that's the case, we should (for now) just go ahead and continue using text and having separate baselines. The per-test cost is relatively low and the number of tests that fall into this bucket is hopefully small? All tests benefit from being obviously self-describing. I've had to debug failing tests that didn't describe what they were testing in words, and it's pretty hard a lot of the time to tell what is even supposed to be happening. Maybe we can come up with a new form of test that puts the explanation of the test somewhere that will not be dumped and the test in a frame, and have the pixel tests dump only the frame. That would also work. I like Darin's idea. Perhaps the dumpAsPixels call could take a frame argument. It's really handy if tests can be self-describing in the browser. Regards, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Pixel tests and displaying text
On Dec 8, 2010, at 2:03 PM, Darin Adler wrote: On Dec 8, 2010, at 1:23 PM, Simon Fraser wrote: On Dec 8, 2010, at 12:57 PM, Darin Adler wrote: On Dec 8, 2010, at 12:13 PM, Peter Kasting wrote: We could greatly decrease the number of these baselines by following a simple rule: don't display text unless you need to. Seems like a good idea. I’m worried a bit, though, that if we can’t use any text in them at all, the tests are then not at all self explanatory. You have to be an expert on the test to understand what it’s testing and what success and failure look like. Do HTML comments fulfill this? Peter, Simon, You both have convinced me that HTML comments could be fine for explaining what a test is testing. Maybe we could come up with a format that makes it likely we’ll spot those comments. The success vs. failure aspects can probably be addressed by the “see no red” and other such things. Maybe we could come up with a way to print the explanation in the browser only, and not in DumpRenderTree. A simple convention could be: if (!window.layoutTestController) document.write(Explanation of what the test is testing goes here.); This is slightly more convenient than an HTML comment when you open a test in the browser to see what's going on. Regards, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Pixel tests and displaying text
On Wed, Dec 8, 2010 at 3:24 PM, Maciej Stachowiak m...@apple.com wrote: Maybe we could come up with a way to print the explanation in the browser only, and not in DumpRenderTree. A simple convention could be: if (!window.layoutTestController) document.write(Explanation of what the test is testing goes here.); This is slightly more convenient than an HTML comment when you open a test in the browser to see what's going on. At least when I've been tracking down test failures, the most convenient thing is the pixel output itself, which I have in both raw and diff form in the various Chromium dashboards. At the point when I need to manually open the test in some fashion, text displayed in the document versus text appearing in the source code are about equally accessible -- if anything, the latter is better for me (opening text editors is easy). Really, I suspect that all that really matters is that we establish some convention, so that I don't have to try opening the test outside the testing environment, reading the source, etc., all in hope of an explanation that I'm not sure exists. PK ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
