Re: [webkit-dev] generic test expectations?

2012-11-13 Thread Dirk Pranke
On Tue, Nov 13, 2012 at 1:04 PM, Tony Chang wrote: > On Tue, Nov 13, 2012 at 12:09 PM, Glenn Adams wrote: >> >> On Tue, Nov 13, 2012 at 1:02 PM, Tony Chang wrote: >>> >>> I don't think we should support port specific ref test results. That >>> kind of misses the point of using a ref test in the

Re: [webkit-dev] generic test expectations?

2012-11-13 Thread Tony Chang
On Tue, Nov 13, 2012 at 12:09 PM, Glenn Adams wrote: > On Tue, Nov 13, 2012 at 1:02 PM, Tony Chang wrote: > >> I don't think we should support port specific ref test results. That >> kind of misses the point of using a ref test in the first place. I mean, >> you may as well check in port speci

Re: [webkit-dev] generic test expectations?

2012-11-13 Thread Dirk Pranke
On Tue, Nov 13, 2012 at 12:16 PM, Tony Chang wrote: > On Tue, Nov 13, 2012 at 12:04 PM, Darin Adler wrote: >> >> On Nov 13, 2012, at 12:02 PM, Tony Chang wrote: >> >> > I don't think we should support port specific ref test results. That >> > kind of misses the point of using a ref test in the f

Re: [webkit-dev] generic test expectations?

2012-11-13 Thread Darin Adler
On Nov 13, 2012, at 12:16 PM, Tony Chang wrote: > Here is a ref test result from ietestcenter: > http://trac.webkit.org/browser/trunk/LayoutTests/ietestcenter/css3/flexbox/flexbox-flex-002-expected.htm > > Looking at that HTML file, it's not immediately obvious that the result is > correct. If

Re: [webkit-dev] generic test expectations?

2012-11-13 Thread Dirk Pranke
On Tue, Nov 13, 2012 at 12:02 PM, Tony Chang wrote: > On Tue, Nov 13, 2012 at 11:35 AM, Dirk Pranke wrote: >> >> On Tue, Nov 13, 2012 at 11:29 AM, Glenn Adams wrote: >> > >> > On Tue, Nov 13, 2012 at 12:09 PM, Dirk Pranke >> > wrote: >> >> >> >> We don't currently support port-specific reftests

Re: [webkit-dev] generic test expectations?

2012-11-13 Thread Tony Chang
On Tue, Nov 13, 2012 at 12:04 PM, Darin Adler wrote: > On Nov 13, 2012, at 12:02 PM, Tony Chang wrote: > > > I don't think we should support port specific ref test results. That > kind of misses the point of using a ref test in the first place. I mean, > you may as well check in port specific pi

Re: [webkit-dev] generic test expectations?

2012-11-13 Thread Glenn Adams
On Tue, Nov 13, 2012 at 1:02 PM, Tony Chang wrote: > On Tue, Nov 13, 2012 at 11:35 AM, Dirk Pranke wrote: > >> On Tue, Nov 13, 2012 at 11:29 AM, Glenn Adams wrote: >> > >> > On Tue, Nov 13, 2012 at 12:09 PM, Dirk Pranke >> wrote: >> >> >> >> We don't currently support port-specific reftests (or

Re: [webkit-dev] generic test expectations?

2012-11-13 Thread Darin Adler
On Nov 13, 2012, at 12:02 PM, Tony Chang wrote: > I don't think we should support port specific ref test results. That kind of > misses the point of using a ref test in the first place. I mean, you may as > well check in port specific pixel results which are easier to review for > correctness.

Re: [webkit-dev] generic test expectations?

2012-11-13 Thread Tony Chang
On Tue, Nov 13, 2012 at 11:35 AM, Dirk Pranke wrote: > On Tue, Nov 13, 2012 at 11:29 AM, Glenn Adams wrote: > > > > On Tue, Nov 13, 2012 at 12:09 PM, Dirk Pranke > wrote: > >> > >> We don't currently support port-specific reftests (or at least, not > >> very well), and we certainly should be tr

Re: [webkit-dev] generic test expectations?

2012-11-13 Thread Dirk Pranke
Right. On Tue, Nov 13, 2012 at 11:36 AM, Darin Adler wrote: > If we do add base test expectations shared by all platforms, please don’t put > the file into LayoutTests/platform/generic; just put it at the top level of > LayoutTests. > > -- Darin > ___

Re: [webkit-dev] generic test expectations?

2012-11-13 Thread Darin Adler
If we do add base test expectations shared by all platforms, please don’t put the file into LayoutTests/platform/generic; just put it at the top level of LayoutTests. -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.

Re: [webkit-dev] generic test expectations?

2012-11-13 Thread Dirk Pranke
On Tue, Nov 13, 2012 at 11:29 AM, Glenn Adams wrote: > > On Tue, Nov 13, 2012 at 12:09 PM, Dirk Pranke wrote: >> >> We don't currently support port-specific reftests (or at least, not >> very well), and we certainly should be trying to minimize where they >> occur. > > > Hmm, I actually used port

Re: [webkit-dev] generic test expectations?

2012-11-13 Thread Glenn Adams
On Tue, Nov 13, 2012 at 12:09 PM, Dirk Pranke wrote: > We don't currently support port-specific reftests (or at least, not > very well), and we certainly should be trying to minimize where they > occur. Hmm, I actually used port specific reftest expectation files in a recent patch [1] (since ro

Re: [webkit-dev] generic test expectations?

2012-11-13 Thread Ryosuke Niwa
On Tue, Nov 13, 2012 at 11:02 AM, Glenn Adams wrote: > That would seem to work only if the test(s) fail the same way on all > ports. In the case that I'm working from, I'm using reftests, where I know > the correct expectations, but the actual behavior will (does) differ on > different ports (whe

Re: [webkit-dev] generic test expectations?

2012-11-13 Thread Dirk Pranke
We don't currently support port-specific reftests (or at least, not very well), and we certainly should be trying to minimize where they occur. Perhaps we should discuss why you need them (in a separate thread with a separate subject line)? It sounds like this largely has to do with what features a

Re: [webkit-dev] generic test expectations?

2012-11-13 Thread Glenn Adams
On Tue, Nov 13, 2012 at 12:02 PM, Glenn Adams wrote: > implements the expected behavior (on some but not all ports), and the > comment further follow-on > s/the comment further/then commit further/ ___ webkit-dev mailing list webkit-dev@lists.webkit.or

Re: [webkit-dev] generic test expectations?

2012-11-13 Thread Glenn Adams
That would seem to work only if the test(s) fail the same way on all ports. In the case that I'm working from, I'm using reftests, where I know the correct expectations, but the actual behavior will (does) differ on different ports (when the corresponding feature is committed). I would like to be

Re: [webkit-dev] generic test expectations?

2012-11-13 Thread Dirk Pranke
That said, I think adding a generic TestExpectations file is a good idea; it would allow us to replace the "-disabled" convention for some tests and allow us to skip tests that were temporarily crashing or timing out everywhere (which *-expected or *-failing wouldn't help with). It should be easy e

Re: [webkit-dev] generic test expectations?

2012-11-13 Thread Ryosuke Niwa
It is customary to add a failing test expectation (i.e. *-expected.txt file that contains the said failure) in such cases. On Tue, Nov 13, 2012 at 10:49 AM, Glenn Adams wrote: > Just checking, but I don't see a way to add test expectations that apply > generically (to all ports). > > It would be

Re: [webkit-dev] generic test expectations?

2012-11-13 Thread Simon Fraser
Just put the expected files in the same directory as the test. Simon On Nov 13, 2012, at 10:49 AM, Glenn Adams wrote: > Just checking, but I don't see a way to add test expectations that apply > generically (to all ports). > > It would be nice to have something like > LayoutTests/platform/ge

[webkit-dev] generic test expectations?

2012-11-13 Thread Glenn Adams
Just checking, but I don't see a way to add test expectations that apply generically (to all ports). It would be nice to have something like LayoutTests/platform/generic/TestExpectations to which one could add new tests that are known to fail everywhere (e.g., because the code that implements a fe