Re: [webkit-dev] Ordering bugmail in committers.py
I think it's also to fix these for other people as you encounter them. FWIW, it's not just the autocomplete code that makes this assumption, so fixing this is more generally useful. On Mon, Oct 11, 2010 at 7:05 AM, Antonio Gomes toniki...@gmail.com wrote: Hi. The autocompletion feature on bugs.webkit.org is simply great. But for it to work even better it is needed that all contributors listed in committers.py (reviewers and committers) properly make their bugmail (email subscribed to bugs.webkit.org) to be the first one in the list. As a illustration of the situation here we go two example of failing CC additions via the autocompletion feature due to the wrong ordering: Martin Robinson and Adam Treat (tr...@kde.org is the first email listed but atr...@rim.com, the third listed, is the bugmail). It would be great if each contributor could fix its own ordering, so the autocompletion works nicely for everyone. Thanks, -- --Antonio Gomes ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Pixel test experiment
My experience is that having a non-zero tolerance makes maintaining the pixel results *harder*. It makes it easier at first of course. But as more and more tests only pass with a non-zero tolerance, it gets harder to figure out if your change causes a regression (e.g. your change causes a pixel test to fail, but when you look at the diff, it includes more changes than you would expect from your patch). Having no tolerance is a pain for sure, but it's much more black and white and thus, it's usually much easier to reason about the correctness of a change. Ojan On Tue, Oct 12, 2010 at 1:43 PM, James Robinson jam...@google.com wrote: To add a concrete data point, http://trac.webkit.org/changeset/69517 caused a number of SVG tests to fail. It required 14 text rebaselines for Mac and a further two more for Leopard (done by Adam Barth). In order to pass the pixel tests in Chromium, it required 1506 new pixel baselines (checked in by the very brave Albert Wong, http://trac.webkit.org/changeset/69543). None of the rebaselining was done by the patch authors and in general I would not expect a patch author that didn't work in Chromium to be expected to update Chromium-specific baselines. I'm a little skeptical of the claim that all SVG changes are run through the pixel tests given that to date none of the affected platform/mac SVG pixel baselines have been updated. This sort of mass-rebaselining is required fairly regularly for minor changes in SVG and in other parts of the codebase. I'd really like for the bots to run the pixel tests on every run, preferably with 0 tolerance. We catch a lot of regressions by running these tests on the Chromium bots that would probably otherwise go unnoticed. However there is a large maintenance cost associated with this coverage. We normally have two engineers (one in PST, one elsewhere in the world) who watch the Chromium bots to triage, suppress, and rebaseline tests as churn is introduced. Questions: - If the pixel tests were running either with a tolerance of 0 or 0.1, what would the expectation be for a patch like http://trac.webkit.org/changeset/69517 which requires hundreds of pixel rebaselines? Would the patch author be expected to update the baselines for the platform/mac port, or would someone else? Thus far the Chromium folks have been the only ones actively maintaining the pixel baselines - which I think is entirely reasonable since we're the only ones trying to run the pixel tests on bots. - Do we have the tools and infrastructure needed to do mass rebaselines in WebKit currently? We've built a number of tools to deal with the Chromium expectations, but since this has been a need unique to Chromium so far the tools only work for Chromium. - James On Fri, Oct 8, 2010 at 11:18 PM, Nikolas Zimmermann zimmerm...@physik.rwth-aachen.de wrote: Am 08.10.2010 um 20:14 schrieb Jeremy Orlow: I'm not an expert on Pixel tests, but my understanding is that in Chromium (where we've always run with tolerance 0) we've seen real regressions that would have slipped by with something like tolerance 0.1. When you have 0 tolerance, it is more maintenance work, but if we can avoid regressions, it seems worth it. Well, that's why I initially argued for tolerance 0. Especially in SVG we had lots of regressions in the past that were below the 0.1 tolerance. I fully support --tolerance 0 as default. Dirk me are also willing to investigate possible problem sources and minimize them. Reftests as Simon said, are a great thing, but it won't help with official test suites like the W3C one - it would be a huge amount of work to create reftests for all of these... Cheers, Niko ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] pixel tests and --tolerance (was Re: Pixel test experiment)
Simon, are you suggesting that we should only use pixel results for ref tests? If not, then we still need to come to a conclusion on this tolerance issue. Dirk, implementing --tolerance in NRWT isn't that hard is it? Getting rid of --tolerance will be a lot of work of making sure all the pixel results that currently pass also pass with --tolerance=0. While I would support someone doing that work, I don't think we should block moving to NRWT on it. Ojan On Fri, Oct 8, 2010 at 1:03 PM, Simon Fraser simon.fra...@apple.com wrote: I think the best solution to this pixel matching problem is ref tests. How practical would it be to use ref tests for SVG? Simon On Oct 8, 2010, at 12:43 PM, Dirk Pranke wrote: Jeremy is correct; the Chromium port has seen real regressions that virtually no concept of a fuzzy match that I can imagine would've caught. new-run-webkit-tests doesn't currently support the tolerance concept at al, and I am inclined to argue that it shouldn't. However, I frequently am wrong about things, so it's quite possible that there are good arguments for supporting it that I'm not aware of. I'm not particularly interested in working on a tool that doesn't do what the group wants it to do, and I would like all of the other WebKit ports to be running pixel tests by default (and new-run-webkit-tests ;) ) since I think it catches bugs. As far as I know, the general sentiment on the list has been that we should be running pixel tests by default, and the reason that we aren't is largely due to the work involved in getting them back up to date and keeping them up to date. I'm sure that fuzzy matching reduces the work load, especially for the sort of mismatches caused by differences in the text antialiasing. In addition, I have heard concerns that we'd like to keep fuzzy matching because people might potentially get different results on machines with different hardware configurations, but I don't know that we have any confirmed cases of that (except for arguably the case of different code paths for gpu-accelerated rendering vs. unaccelerated rendering). If we made it easier to maintain the baselines (improved tooling like the chromium's rebaselining tool, add reftest support, etc.) are there still compelling reasons for supporting --tolerance -based testing as opposed to exact matching? -- Dirk On Fri, Oct 8, 2010 at 11:14 AM, Jeremy Orlow jor...@chromium.org wrote: I'm not an expert on Pixel tests, but my understanding is that in Chromium (where we've always run with tolerance 0) we've seen real regressions that would have slipped by with something like tolerance 0.1. When you have 0 tolerance, it is more maintenance work, but if we can avoid regressions, it seems worth it. J On Fri, Oct 8, 2010 at 10:58 AM, Nikolas Zimmermann zimmerm...@physik.rwth-aachen.de wrote: Am 08.10.2010 um 19:53 schrieb Maciej Stachowiak: On Oct 8, 2010, at 12:46 AM, Nikolas Zimmermann wrote: Am 08.10.2010 um 00:44 schrieb Maciej Stachowiak: On Oct 7, 2010, at 6:34 AM, Nikolas Zimmermann wrote: Good evening webkit folks, I've finished landing svg/ pixel test baselines, which pass with --tolerance 0 on my 10.5 10.6 machines. As the pixel testing is very important for the SVG tests, I'd like to run them on the bots, experimentally, so we can catch regressions easily. Maybe someone with direct access to the leopard snow leopard bots, could just run run-webkit-tests --tolerance 0 -p svg and mail me the results? If it passes, we could maybe run the pixel tests for the svg/ subdirectory on these bots? Running pixel tests would be great, but can we really expect the results to be stable cross-platform with tolerance 0? Perhaps we should start with a higher tolerance level. Sure, we could do that. But I'd really like to get a feeling, for what's problematic first. If we see 95% of the SVG tests pass with --tolerance 0, and only a few need higher tolerances (64bit vs. 32bit aa differences, etc.), I could come up with a per-file pixel test tolerance extension to DRT, if it's needed. How about starting with just one build slave (say. Mac Leopard) that runs the pixel tests for SVG, with --tolerance 0 for a while. I'd be happy to identify the problems, and see if we can make it work, somehow :-) The problem I worry about is that on future Mac OS X releases, rendering of shapes may change in some tiny way that is not visible but enough to cause failures at tolerance 0. In the past, such false positives arose from time to time, which is one reason we added pixel test tolerance in the first place. I don't think running pixel tests on just one build slave will help us understand that risk. I think we'd just update the baseline to the newer OS X release, then, like it has been done for the tiger - leopard, leopard - snow leopard
[webkit-dev] Buildbot Performance
I am in the process of moving buildbot onto faster storage which should help with performance. However, during the move, performance will be even worse due to the extra i/o. There will be a downtime period in the next few days to do the final switchover, but I won't know when that will be until the preliminary copying is done. I am trying not to kill the master completely, but there have been some slave disconnects due to the load already this morning. I'll let everyone know when the downtime will be once I know. Thanks, -Bill ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] pixel tests and --tolerance (was Re: Pixel test experiment)
I'm not sure if this could be make to work with SVG (might require some additions to LayoutTestController), but Philip Taylor's canvas test suite (in LayoutTests/canvas/philip) compares pixels programmatically in JavaScript. This has the major advantage that it doesn't require pixel results, and allows for a per-test level of fuzziness/tolerance (if required). Obviously we would still want to have some tests remain pixel tests, as these tests only cover a subset of pixels, but it might be a good alternative to consider when writing new tests (especially for regressions, where a single pixel correctly chosen can often correctly isolate the problem). Stephen On Thu, Oct 14, 2010 at 12:06 PM, Ojan Vafai o...@chromium.org wrote: Simon, are you suggesting that we should only use pixel results for ref tests? If not, then we still need to come to a conclusion on this tolerance issue. Dirk, implementing --tolerance in NRWT isn't that hard is it? Getting rid of --tolerance will be a lot of work of making sure all the pixel results that currently pass also pass with --tolerance=0. While I would support someone doing that work, I don't think we should block moving to NRWT on it. Ojan On Fri, Oct 8, 2010 at 1:03 PM, Simon Fraser simon.fra...@apple.comwrote: I think the best solution to this pixel matching problem is ref tests. How practical would it be to use ref tests for SVG? Simon On Oct 8, 2010, at 12:43 PM, Dirk Pranke wrote: Jeremy is correct; the Chromium port has seen real regressions that virtually no concept of a fuzzy match that I can imagine would've caught. new-run-webkit-tests doesn't currently support the tolerance concept at al, and I am inclined to argue that it shouldn't. However, I frequently am wrong about things, so it's quite possible that there are good arguments for supporting it that I'm not aware of. I'm not particularly interested in working on a tool that doesn't do what the group wants it to do, and I would like all of the other WebKit ports to be running pixel tests by default (and new-run-webkit-tests ;) ) since I think it catches bugs. As far as I know, the general sentiment on the list has been that we should be running pixel tests by default, and the reason that we aren't is largely due to the work involved in getting them back up to date and keeping them up to date. I'm sure that fuzzy matching reduces the work load, especially for the sort of mismatches caused by differences in the text antialiasing. In addition, I have heard concerns that we'd like to keep fuzzy matching because people might potentially get different results on machines with different hardware configurations, but I don't know that we have any confirmed cases of that (except for arguably the case of different code paths for gpu-accelerated rendering vs. unaccelerated rendering). If we made it easier to maintain the baselines (improved tooling like the chromium's rebaselining tool, add reftest support, etc.) are there still compelling reasons for supporting --tolerance -based testing as opposed to exact matching? -- Dirk On Fri, Oct 8, 2010 at 11:14 AM, Jeremy Orlow jor...@chromium.org wrote: I'm not an expert on Pixel tests, but my understanding is that in Chromium (where we've always run with tolerance 0) we've seen real regressions that would have slipped by with something like tolerance 0.1. When you have 0 tolerance, it is more maintenance work, but if we can avoid regressions, it seems worth it. J On Fri, Oct 8, 2010 at 10:58 AM, Nikolas Zimmermann zimmerm...@physik.rwth-aachen.de wrote: Am 08.10.2010 um 19:53 schrieb Maciej Stachowiak: On Oct 8, 2010, at 12:46 AM, Nikolas Zimmermann wrote: Am 08.10.2010 um 00:44 schrieb Maciej Stachowiak: On Oct 7, 2010, at 6:34 AM, Nikolas Zimmermann wrote: Good evening webkit folks, I've finished landing svg/ pixel test baselines, which pass with --tolerance 0 on my 10.5 10.6 machines. As the pixel testing is very important for the SVG tests, I'd like to run them on the bots, experimentally, so we can catch regressions easily. Maybe someone with direct access to the leopard snow leopard bots, could just run run-webkit-tests --tolerance 0 -p svg and mail me the results? If it passes, we could maybe run the pixel tests for the svg/ subdirectory on these bots? Running pixel tests would be great, but can we really expect the results to be stable cross-platform with tolerance 0? Perhaps we should start with a higher tolerance level. Sure, we could do that. But I'd really like to get a feeling, for what's problematic first. If we see 95% of the SVG tests pass with --tolerance 0, and only a few need higher tolerances (64bit vs. 32bit aa differences, etc.), I could come up with a per-file pixel test tolerance extension to DRT, if it's needed. How about starting with just one build
Re: [webkit-dev] pixel tests and --tolerance (was Re: Pixel test experiment)
On Oct 14, 2010, at 9:06 AM, Ojan Vafai wrote: Simon, are you suggesting that we should only use pixel results for ref tests? In an ideal world, yes. But we have such a huge body of existing tests that converting them all to ref tests is a non-starter, so I agree that we need to resolve the tolerance issue. However, at some point I'd like to see us get to a stage where new pixel tests must be ref tests. Simon If not, then we still need to come to a conclusion on this tolerance issue. Dirk, implementing --tolerance in NRWT isn't that hard is it? Getting rid of --tolerance will be a lot of work of making sure all the pixel results that currently pass also pass with --tolerance=0. While I would support someone doing that work, I don't think we should block moving to NRWT on it. Ojan ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Buildbot Performance
On Oct 14, 2010, at 9:27 AM, William Siegrist wrote: I am in the process of moving buildbot onto faster storage which should help with performance. However, during the move, performance will be even worse due to the extra i/o. There will be a downtime period in the next few days to do the final switchover, but I won't know when that will be until the preliminary copying is done. I am trying not to kill the master completely, but there have been some slave disconnects due to the load already this morning. I'll let everyone know when the downtime will be once I know. The copying of data will take days at the rate we're going, and the server is exhibiting some strange memory paging in the process. I am going to reboot the server and try copying with the buildbot master down. The master will be down for about 15m, if I can't get the copy done in that time I will schedule a longer downtime at a better time. Sorry for the churn. -Bill ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] Selection Highlight and MathML
I've taken an initial look at this bug: https://bugs.webkit.org/show_bug.cgi?id=43818 and I'm curious as to why MathML seems to be treated differently for selection highlights. Is there something a rendering object should do to define the selection highlight? Ideally, we'd just have one highlight for the root MathML element's rendering object (typically, 'math'). -- --Alex Milowski The excellence of grammar as a guide is proportional to the paucity of the inflexions, i.e. to the degree of analysis effected by the language considered. Bertrand Russell in a footnote of Principles of Mathematics ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] pixel tests and --tolerance (was Re: Pixel test experiment)
On Thu, Oct 14, 2010 at 9:06 AM, Ojan Vafai o...@chromium.org wrote: Dirk, implementing --tolerance in NRWT isn't that hard is it? Getting rid of --tolerance will be a lot of work of making sure all the pixel results that currently pass also pass with --tolerance=0. While I would support someone doing that work, I don't think we should block moving to NRWT on it. Assuming we implement it only for the ports that currently use tolerance on old-run-webkit-tests, no, I wouldn't expect it to be hard. Dunno how much work it would be to implement tolerance on the chromium image_diff implementations (side note: it would be nice if these binaries weren't port-specific, but that's another topic). As to how many files we'd have to rebaseline for the base ports, I don't know how many there are compared to how many fail pixel tests, period. I'll run a couple tests and find out. -- Dirk Ojan On Fri, Oct 8, 2010 at 1:03 PM, Simon Fraser simon.fra...@apple.com wrote: I think the best solution to this pixel matching problem is ref tests. How practical would it be to use ref tests for SVG? Simon On Oct 8, 2010, at 12:43 PM, Dirk Pranke wrote: Jeremy is correct; the Chromium port has seen real regressions that virtually no concept of a fuzzy match that I can imagine would've caught. new-run-webkit-tests doesn't currently support the tolerance concept at al, and I am inclined to argue that it shouldn't. However, I frequently am wrong about things, so it's quite possible that there are good arguments for supporting it that I'm not aware of. I'm not particularly interested in working on a tool that doesn't do what the group wants it to do, and I would like all of the other WebKit ports to be running pixel tests by default (and new-run-webkit-tests ;) ) since I think it catches bugs. As far as I know, the general sentiment on the list has been that we should be running pixel tests by default, and the reason that we aren't is largely due to the work involved in getting them back up to date and keeping them up to date. I'm sure that fuzzy matching reduces the work load, especially for the sort of mismatches caused by differences in the text antialiasing. In addition, I have heard concerns that we'd like to keep fuzzy matching because people might potentially get different results on machines with different hardware configurations, but I don't know that we have any confirmed cases of that (except for arguably the case of different code paths for gpu-accelerated rendering vs. unaccelerated rendering). If we made it easier to maintain the baselines (improved tooling like the chromium's rebaselining tool, add reftest support, etc.) are there still compelling reasons for supporting --tolerance -based testing as opposed to exact matching? -- Dirk On Fri, Oct 8, 2010 at 11:14 AM, Jeremy Orlow jor...@chromium.org wrote: I'm not an expert on Pixel tests, but my understanding is that in Chromium (where we've always run with tolerance 0) we've seen real regressions that would have slipped by with something like tolerance 0.1. When you have 0 tolerance, it is more maintenance work, but if we can avoid regressions, it seems worth it. J On Fri, Oct 8, 2010 at 10:58 AM, Nikolas Zimmermann zimmerm...@physik.rwth-aachen.de wrote: Am 08.10.2010 um 19:53 schrieb Maciej Stachowiak: On Oct 8, 2010, at 12:46 AM, Nikolas Zimmermann wrote: Am 08.10.2010 um 00:44 schrieb Maciej Stachowiak: On Oct 7, 2010, at 6:34 AM, Nikolas Zimmermann wrote: Good evening webkit folks, I've finished landing svg/ pixel test baselines, which pass with --tolerance 0 on my 10.5 10.6 machines. As the pixel testing is very important for the SVG tests, I'd like to run them on the bots, experimentally, so we can catch regressions easily. Maybe someone with direct access to the leopard snow leopard bots, could just run run-webkit-tests --tolerance 0 -p svg and mail me the results? If it passes, we could maybe run the pixel tests for the svg/ subdirectory on these bots? Running pixel tests would be great, but can we really expect the results to be stable cross-platform with tolerance 0? Perhaps we should start with a higher tolerance level. Sure, we could do that. But I'd really like to get a feeling, for what's problematic first. If we see 95% of the SVG tests pass with --tolerance 0, and only a few need higher tolerances (64bit vs. 32bit aa differences, etc.), I could come up with a per-file pixel test tolerance extension to DRT, if it's needed. How about starting with just one build slave (say. Mac Leopard) that runs the pixel tests for SVG, with --tolerance 0 for a while. I'd be happy to identify the problems, and see if we can make it work, somehow :-) The problem I worry about is that on future Mac OS X releases, rendering of shapes may change in some tiny
Re: [webkit-dev] Selection Highlight and MathML
On Oct 14, 2010, at 2:12 PM, Alex Milowski wrote: I'm curious as to why MathML seems to be treated differently for selection highlights. Differently from what? I’m not sure what your question is. -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Selection Highlight and MathML
On Thu, Oct 14, 2010 at 5:19 PM, Darin Adler da...@apple.com wrote: On Oct 14, 2010, at 2:12 PM, Alex Milowski wrote: I'm curious as to why MathML seems to be treated differently for selection highlights. Differently from what? I’m not sure what your question is. If you try out the attachment from the bug you'll see what I mean. Essentially, MathML seems to get several overlapping highlights that get darker and darker the more that they overlap via several layers. As such, the content eventually gets washed out by the highlight color. I do not see this behavior for non-MathML content and that leads me to believe we need to something in the rendering objects for MathML. -- --Alex Milowski The excellence of grammar as a guide is proportional to the paucity of the inflexions, i.e. to the degree of analysis effected by the language considered. Bertrand Russell in a footnote of Principles of Mathematics ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
