Re: [dev-servo] Categorizing Stylo reftest failures
On 8/10/17 2:25 PM, Chris Peterson wrote: Besides fixing the regressions, we should review the test cases where Gecko fails but Stylo passes to make sure we're not unintentionally shipping new CSS features. Indeed. And any remaining such should probably be annotated with Gecko bugs or otherwise noted that we've looked at them. -Boris ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] Categorizing Stylo reftest failures
On 2017-08-10 1:55 AM, James Graham wrote: On 10/08/17 02:53, Manish Goregaokar wrote: https://gist.github.com/Manishearth/086118c940ff86a6cfc573f53c508279 FWIW I made a list of the differences between reftest results in web-platform-tests with stylo enabled vs not (based on the expectation metadata files). Paths are relative to the root of web-platform-tests. https://pastebin.mozilla.org/9029321 Thanks, James. We have a Stylo meta-bug for web-platform-test issues. I filed a new bug with the particular issues you reported: https://bugzilla.mozilla.org/show_bug.cgi?id=1389187 Besides fixing the regressions, we should review the test cases where Gecko fails but Stylo passes to make sure we're not unintentionally shipping new CSS features. chris ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] Categorizing Stylo reftest failures
On 10/08/17 02:53, Manish Goregaokar wrote: https://gist.github.com/Manishearth/086118c940ff86a6cfc573f53c508279 FWIW I made a list of the differences between reftest results in web-platform-tests with stylo enabled vs not (based on the expectation metadata files). Paths are relative to the root of web-platform-tests. https://pastebin.mozilla.org/9029321 ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] Categorizing Stylo reftest failures
Great work Manish! On Wed, Aug 9, 2017 at 6:53 PM, Manish Goregaokar wrote: > https://gist.github.com/Manishearth/086118c940ff86a6cfc573f53c508279 > > I've now gone through the failures and categorized them as yes/no/maybe > based on whether they should block shipping. I've filed bugs for and > partially investigated all of the "yes" ones (and fixed some of them). Some > of these bugs are unassigned; feel free to pick them up! > > The "maybe" ones are ones which I don't *think* are important but I haven't > had a chance to properly investigate yet. I will do that soon. > > > We're looking pretty good on this front, only a couple of "yes" failures > left. > > There may have been additional failures introduced since I generated this > doc (I removed new passes but did not add new failures), which I also > intend to go through at some point. > > > Thanks, > -Manish > ___ > dev-servo mailing list > dev-servo@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-servo > ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] Categorizing Stylo reftest failures
https://gist.github.com/Manishearth/086118c940ff86a6cfc573f53c508279 I've now gone through the failures and categorized them as yes/no/maybe based on whether they should block shipping. I've filed bugs for and partially investigated all of the "yes" ones (and fixed some of them). Some of these bugs are unassigned; feel free to pick them up! The "maybe" ones are ones which I don't *think* are important but I haven't had a chance to properly investigate yet. I will do that soon. We're looking pretty good on this front, only a couple of "yes" failures left. There may have been additional failures introduced since I generated this doc (I removed new passes but did not add new failures), which I also intend to go through at some point. Thanks, -Manish ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] Categorizing Stylo reftest failures
We aren't! I'm just not seeing that bug in the reftests outside of audio and video, and TYLin is already looking into it anyway. Thanks, -Manish Goregaokar On Mon, Jul 24, 2017 at 8:20 AM, Boris Zbarsky wrote: > On 7/24/17 1:53 AM, Manish Goregaokar wrote: >> >> and the styloVsGecko / issue (which does not block landing >> since it's a >> slight discrepancy between stylo and gecko) > > > Looks like it's a difference in vertical alignment. How sure are we that > this won't affect web pages outside of media controls? > > > -Boris > ___ > dev-servo mailing list > dev-servo@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-servo ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] Categorizing Stylo reftest failures
On 7/24/17 1:53 AM, Manish Goregaokar wrote: and the styloVsGecko / issue (which does not block landing since it's a slight discrepancy between stylo and gecko) Looks like it's a difference in vertical alignment. How sure are we that this won't affect web pages outside of media controls? -Boris ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] Categorizing Stylo reftest failures
Now that we've fixed most of these I went through it again and checked stuff. Filtering out scoped style and web components failures we only have a hundred tests left I recategorized most of them: https://gist.github.com/Manishearth/086118c940ff86a6cfc573f53c508279 The major issues right now are first-line and the styloVsGecko / issue (which does not block landing since it's a slight discrepancy between stylo and gecko) -Manish Goregaokar On Thu, May 11, 2017 at 5:19 PM, Manish Goregaokar wrote: > And, it's done! All reftest failures classified. > > Permalink to current revision: > https://gist.github.com/Manishearth/086118c940ff86a6cfc573f53c508279/a181e327a4e2d83d75e28862d1fc01e121923bff > > Bear in mind that the try run this is based on is a few days old so some of > the things there may be fixed in master. I've been making drive-by patches > for many of the things I come across. > > I'll soon do a separate analysis of the HasAuthorSpecifiedRules stuff to see > what's still failing since Matt's patches have landed. There are 913 tests > marked as HasAuthorSpecified in that file, and Matt's patches fixed 530, so > we still have some input widget bugs floating around. > > > I will also be removing now-passing tests if I see them, but don't rely on > that being true. I don't intend to put too much time into maintaining this > list; the main point was to get an idea of what needs to be fixed. I will > try to investigate things which still fail when their corresponding bug > lands. > > > Thanks, > > -Manish Goregaokar > > On Wed, May 10, 2017 at 5:14 PM, Manish Goregaokar > wrote: >> >> So I took a try push with all expectations set as passing and went about >> categorizing the reftest failures. >> >> I'm halfway done. This is pre-HasAuthorSpecifiedRules and there are lots >> of related failures. At some point I'll extract all the HASR-marked failures >> and have a second look at the ones which are still not passing after the >> HASR bug lands. >> >> https://gist.github.com/Manishearth/086118c940ff86a6cfc573f53c508279 >> >> While I've investigated many of these, many others still don't have well >> known reasons. >> >> It would be quite helpful if everyone could help investigate. Ctrl-F >> "unknown" on that page, anything with "unknown" in the explanation needs >> more investigation. >> >> If you do file bugs (or find existing bugs), please let me know so that I >> can update the page. I haven't yet had time to really file bugs for each >> failure, and I intend to do this once done with the rest, but help now would >> be appreciated! >> >> There are also some possibly-low-hanging-fruit failures where I've >> investigated the reason and mentioned it, but not yet filed an issue. >> >> I'll continue to push more reftest failure categorizations as I get >> through, and I intend to write some scripts that let me keep this doc >> somewhat up to date as major groups of failures get fixed. >> >> Thanks, >> -Manish Goregaokar > > ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] Categorizing Stylo reftest failures
And, it's done! All reftest failures classified. Permalink to current revision: https://gist.github.com/Manishearth/086118c940ff86a6cfc573f53c508279/a181e327a4e2d83d75e28862d1fc01e121923bff Bear in mind that the try run this is based on is a few days old so some of the things there may be fixed in master. I've been making drive-by patches for many of the things I come across. I'll soon do a separate analysis of the HasAuthorSpecifiedRules stuff to see what's still failing since Matt's patches have landed. There are 913 tests marked as HasAuthorSpecified in that file, and Matt's patches fixed 530, so we still have some input widget bugs floating around. I will also be removing now-passing tests if I see them, but don't rely on that being true. I don't intend to put too much time into maintaining this list; the main point was to get an idea of what needs to be fixed. I will try to investigate things which still fail when their corresponding bug lands. Thanks, -Manish Goregaokar On Wed, May 10, 2017 at 5:14 PM, Manish Goregaokar wrote: > So I took a try push with all expectations set as passing and went about > categorizing the reftest failures. > > I'm halfway done. This is pre-HasAuthorSpecifiedRules and there are lots > of related failures. At some point I'll extract all the HASR-marked > failures and have a second look at the ones which are still not passing > after the HASR bug lands. > > https://gist.github.com/Manishearth/086118c940ff86a6cfc573f53c508279 > > While I've investigated many of these, many others still don't have well > known reasons. > > It would be quite helpful if everyone could help investigate. Ctrl-F > "unknown" on that page, anything with "unknown" in the explanation needs > more investigation. > > If you do file bugs (or find existing bugs), please let me know so that I > can update the page. I haven't yet had time to really file bugs for each > failure, and I intend to do this once done with the rest, but help now > would be appreciated! > > There are also some possibly-low-hanging-fruit failures where I've > investigated the reason and mentioned it, but not yet filed an issue. > > I'll continue to push more reftest failure categorizations as I get > through, and I intend to write some scripts that let me keep this doc > somewhat up to date as major groups of failures get fixed. > > Thanks, > -Manish Goregaokar > ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] Categorizing Stylo reftest failures
First link in the file! https://treeherder.mozilla.org/#/jobs?repo=try&revision=bff0878649f10db8946d50e760b0d13f0f3be9be Also, each section in the file already links to the analyzer results for that subset of the tests. -Manish Goregaokar ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] Categorizing Stylo reftest failures
On 5/10/17 8:14 PM, Manish Goregaokar wrote: So I took a try push with all expectations set as passing and went about categorizing the reftest failures. Do you mind linking to the try push? Looking at the actual reftest-analyzer results is helpful for categorizing things. -Boris ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
