Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
Here are some examples of old and new format: BUGCR24182 SLOW MAC DEBUG : fast/css/large-list-of-rules-crash.html = PASS BUGCR24182 SLOW LINUX MAC DEBUG : fast/dom/Window/window-postmessage-clone-really-deep-array.html = PASS BUGCR24182 SLOW MAC DEBUG : fast/forms/select-set-length-with-mutation-remove.html = PASS crbug.com/24182 [Mac Debug] fast/css/large-list-of-rules-crash.html [Slow] crbug.com/24182 [Linux Mac Debug] fast/dom/Window/window-postmessage-clone-really-deep-array.html [Slow] crbug.com/24182 [Mac Debug] fast/dom/Window/window-postmessage-clone-really-deep-array.html [Slow] BUGWK87364 DEBUG : fast/dom/shadow/drop-event-for-input-in-shadow.html = TEXT PASS BUGWK87364 DEBUG : fast/dom/shadow/drop-event-in-shadow.html = TEXT PASS webkit.org/b/87364 [Debug] fast/dom/shadow/drop-event-for-input-in-shadow.html [Text Pass] webkit.org/b/87364 [Debug] fast/dom/shadow/drop-event-in-shadow.html [Text Pass] BUG_EAE SKIP : fast/repaint/reflection-repaint-test.html = PASS BUG_EAE SKIP : fast/repaint/transform-layout-repaint.html = PASS Bug(eae) fast/repaint/reflection-repaint-test.html [Skip] Bug(eae) fast/repaint/transform-layout-repaint.html [Skip] I've started to think that we might want to keep WontFix at the beginning of each line as supposed to placing them at the end of line as initially proposed: Beginning of line (current proposal with expectations): WONTFIX SKIP : media/media-captions.html = TIMEOUT BUGWK43459 WONTFIX : http/tests/appcache/origin-quota.html = MISSING TEXT IMAGE+TEXT media/media-captions.html [WontFix] webkit.org/b/43459 WontFix http/tests/appcache/origin-quota.html [WontFix Missing Text Image+Text] End of line: WONTFIX SKIP : media/media-captions.html = TIMEOUT BUGWK43459 WONTFIX : http/tests/appcache/origin-quota.html = MISSING TEXT IMAGE+TEXT WontFix media/media-captions.html WontFix webkit.org/b/43459 http/tests/appcache/origin-quota.html [Missing Text Image+Text] In fact, there are very few entries of WontFix with a bug number at least in Chromium port's test expectations file. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
Oops, some really bad typos. On Thu, Jun 14, 2012 at 3:10 AM, Ryosuke Niwa rn...@webkit.org wrote: Beginning of line (current proposal with expectations): End of line with expectations (current proposal) WONTFIX SKIP : media/media-captions.html = TIMEOUT BUGWK43459 WONTFIX : http/tests/appcache/origin-quota.html = MISSING TEXT IMAGE+TEXT media/media-captions.html [WontFix] webkit.org/b/43459 WontFix http/tests/appcache/origin-quota.html [WontFix Missing Text Image+Text] End of line: Beginning of line WONTFIX SKIP : media/media-captions.html = TIMEOUT BUGWK43459 WONTFIX : http/tests/appcache/origin-quota.html = MISSING TEXT IMAGE+TEXT WontFix media/media-captions.html WontFix webkit.org/b/43459 http/tests/appcache/origin-quota.html [Missing Text Image+Text] In fact, there are very few entries of WontFix with a bug number at least in Chromium port's test expectations file. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] please stop sending messages
Dear, I kindly request to stop the messages, -- kiran Gokidi +91-9043174437 ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Wed, Jun 13, 2012 at 6:53 PM, Dirk Pranke dpra...@chromium.org wrote: * We'll probably rename IMAGE+TEXT to IMAGE_AND_TEXT. Can someone please remind me why IMAGE+TEXT even exists? Wouldn't it be simpler to just mark a test as follows? - IMAGE : allow image failure; go red if there is a text failure - TEXT: allow text failure; go red if there is an image failure - IMAGE TEXT: allow text and/or image failure ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger epo...@chromium.org wrote: Can someone please remind me why IMAGE+TEXT even exists? Wouldn't it be simpler to just mark a test as follows? - IMAGE : allow image failure; go red if there is a text failure - TEXT: allow text failure; go red if there is an image failure - IMAGE TEXT: allow text and/or image failure The distinction is that IMAGE TEXT will allow image, text, or both to fail, thus making transitions among the three generate no events. IMAGE+TEXT says specifically that we expect both to fail and that if one starts passing, someone should do something. (For example, maybe someone checks in a partial rebaseline where they miss the image expectations.) PK ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 1:44 PM, Peter Kasting pkast...@chromium.orgwrote: On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger epo...@chromium.org wrote: Can someone please remind me why IMAGE+TEXT even exists? Wouldn't it be simpler to just mark a test as follows? - IMAGE : allow image failure; go red if there is a text failure - TEXT: allow text failure; go red if there is an image failure - IMAGE TEXT: allow text and/or image failure The distinction is that IMAGE TEXT will allow image, text, or both to fail, thus making transitions among the three generate no events. IMAGE+TEXT says specifically that we expect both to fail and that if one starts passing, someone should do something. (For example, maybe someone checks in a partial rebaseline where they miss the image expectations.) Not to bike-shed on anything, but I think we should rename Text and Image to TextOnly and ImageOnly. Every single person I know, including myself, had never got the distinction between IMAGE TEXT and IMAGE+TEXT without someone explaining it to him/her . - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 4:44 PM, Peter Kasting pkast...@chromium.orgwrote: On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger epo...@chromium.org wrote: Can someone please remind me why IMAGE+TEXT even exists? Wouldn't it be simpler to just mark a test as follows? - IMAGE : allow image failure; go red if there is a text failure - TEXT: allow text failure; go red if there is an image failure - IMAGE TEXT: allow text and/or image failure The distinction is that IMAGE TEXT will allow image, text, or both to fail, thus making transitions among the three generate no events. IMAGE+TEXT says specifically that we expect both to fail and that if one starts passing, someone should do something. (For example, maybe someone checks in a partial rebaseline where they miss the image expectations.) Thanks for the explanation. That makes sense, although it seems to me that the problem of no-events-generated-by-changes-in-actual-images-while-IMAGE-failure-is-expected is about 100x worse for us. But that's not a reason to hide these particular transitions! :-) PK ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 1:44 PM, Peter Kasting pkast...@chromium.org wrote: On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger epo...@chromium.org wrote: Can someone please remind me why IMAGE+TEXT even exists? Wouldn't it be simpler to just mark a test as follows? IMAGE : allow image failure; go red if there is a text failure TEXT: allow text failure; go red if there is an image failure IMAGE TEXT: allow text and/or image failure The distinction is that IMAGE TEXT will allow image, text, or both to fail, thus making transitions among the three generate no events. IMAGE+TEXT says specifically that we expect both to fail and that if one starts passing, someone should do something. (For example, maybe someone checks in a partial rebaseline where they miss the image expectations.) Also, unlike the now-deleted FAIL keyword, IMAGE TEXT indicates that the test is expected to be flaky. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Running layout tests for the chromium port, using the multi-processing architecture and fully sandboxed
I created a page with instructions how to use content_shell here: https://sites.google.com/a/chromium.org/dev/developers/testing/webkit-layout-tests/content-shell At the bottom, I summarized next steps, esp. how we can reuse chromium's test controller for the content shell best -jochen On Mon, Jun 11, 2012 at 1:53 AM, Darin Fisher da...@chromium.org wrote: On Fri, Jun 8, 2012 at 12:51 PM, Jochen Eisinger joc...@chromium.orgwrote: On Fri, Jun 8, 2012 at 9:41 PM, Ryosuke Niwa rn...@webkit.org wrote: On Fri, Jun 8, 2012 at 12:18 PM, Jochen Eisinger joc...@chromium.orgwrote: I've implemented initial support for running layout tests using chromium's content_shell instead of test_shell as the current DRT implementation does. It's still a very experimental, but might already be interesting for some of you to try. This is great! Thanks a lot on working this. 1. Make sure your WebKit is at least at r119852 (see http://trac.webkit.org/wiki/Chromium for prerequisites) 2. Apply the attachment from https://bugs.webkit.org/show_bug.cgi?id=87045 3. In Source/WebKit/chromium run gclient sync 4. build webkit as usual E.g. for a debug build on Linux, this should give you out/Debug/content_shell You can now run layout tests like this: new-run-webkit-tests --chromium --debug --driver-name=content_shell --additional-drt-flag=--dump-render-tree LayoutTests/storage/indexeddb/* You'll notice that not all tests are passing yet, mainly because not all (or actually, almost none of the) layoutTestController features are implemented yet. Where is layoutTestController implemented? We definitely need to move the implementation of layoutTestController, eventSender, etc... into WebKit repository because we often rename functions, etc... in WebKit. It's unacceptable to require having to modify Chromium code in order to do this refactoring in the future. It's currently here: http://code.google.com/searchframe#OAMlx_jo-ck/src/content/shell/layout_test_controller.jsexact_package=chromium Per my other thread about sharing IDLs between DumpRenderTree and WebKitTestRunner, I would like to see us sharing IDL with WebKitTestRunner instead of adding yet another binding code. Another missing feature is producing pixel results. However, I'm currently concentrating on text results, as I think the biggest benefit is the ability to run storage tests on the real storage implementation. That sounds great to me but we definitely need to support pixel results eventually. I'm more than happy to help you on that but that requires the codebase to be moved into WebKit repository. Here's the basic problem: content_shell depends on content, so moving this on the webkit repository would mean that webkit depended on content. Another solution would be to formalize the test shell API our current layout test controller in webkit uses (Tools/DumpRenderTree/chromium), and add a method to chromium's webkit support library that returns a webview that supports all the hooks required. The webview could then either be implemented by test_shell or by content_shell ^^^ This second solution seems like it will work well. It will enable the guts of layoutTestController to remain in the WebKit repository. This is just a variation of exactly what we do today. You only need to move creation of WebView to Chromium so that we can eliminate WebViewHost.cpp (and other simple application shell bits). -Darin best -jochen ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Jun 14, 2012, at 1:47 PM, Ryosuke Niwa rn...@webkit.org wrote:
On Thu, Jun 14, 2012 at 1:44 PM, Peter Kasting pkast...@chromium.org wrote:
On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger epo...@chromium.org wrote:
Can someone please remind me why IMAGE+TEXT even exists?
Wouldn't it be simpler to just mark a test as follows?
IMAGE : allow image failure; go red if there is a text failure
TEXT: allow text failure; go red if there is an image failure
IMAGE TEXT: allow text and/or image failure
The distinction is that IMAGE TEXT will allow image, text, or both to fail,
thus making transitions among the three generate no events. IMAGE+TEXT says
specifically that we expect both to fail and that if one starts passing,
someone should do something. (For example, maybe someone checks in a partial
rebaseline where they miss the image expectations.)
Not to bike-shed on anything, but I think we should rename Text and Image to
TextOnly and ImageOnly. Every single person I know, including myself, had
never got the distinction between IMAGE TEXT and IMAGE+TEXT without someone
explaining it to him/her .
I think IMAGE+TEXT is not a very useful distinction from TEXT either. I checked
for uses of TEXT that is not IMAGE+TEXT in the Chromium TextExpectations, and
it seems that nearly all instances fall into one of the two following
categories:
1) text-only test, so IMAGE+TEXT would not have different semantics from TEXT
(the vast majority)
2) Flaky test that may actually pass, so distinguishing what happens with the
image result is of limited utility (most of these are also text-only tests;
only a small subset even have an image result)
Thus, I think Fail and ImageOnlyFail would be more useful and understandable
categories than {TEXT, IMAGE, TEXT+IMAGE, TEXT IMAGE}. Fail would have the
semantic that a text failure is expected, and image result if any can either
pass or fail.
Regards,
Maciej
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Rename FAIL to DIFF Was (Re: PSA: FAIL test expectation does not encompass MISSING, CRASH, or TIMEOUT)
Done in http://trac.webkit.org/changeset/120371 - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 4:22 PM, Maciej Stachowiak m...@apple.com wrote:
On Jun 14, 2012, at 1:47 PM, Ryosuke Niwa rn...@webkit.org wrote:
On Thu, Jun 14, 2012 at 1:44 PM, Peter Kasting pkast...@chromium.org
wrote:
On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger epo...@chromium.org wrote:
Can someone please remind me why IMAGE+TEXT even exists?
Wouldn't it be simpler to just mark a test as follows?
IMAGE : allow image failure; go red if there is a text failure
TEXT: allow text failure; go red if there is an image failure
IMAGE TEXT: allow text and/or image failure
The distinction is that IMAGE TEXT will allow image, text, or both to
fail, thus making transitions among the three generate no events.
IMAGE+TEXT says specifically that we expect both to fail and that if one
starts passing, someone should do something. (For example, maybe someone
checks in a partial rebaseline where they miss the image expectations.)
Not to bike-shed on anything, but I think we should rename Text and Image to
TextOnly and ImageOnly. Every single person I know, including myself, had
never got the distinction between IMAGE TEXT and IMAGE+TEXT without someone
explaining it to him/her .
I think IMAGE+TEXT is not a very useful distinction from TEXT either. I
checked for uses of TEXT that is not IMAGE+TEXT in the Chromium
TextExpectations, and it seems that nearly all instances fall into one of
the two following categories:
1) text-only test, so IMAGE+TEXT would not have different semantics from
TEXT (the vast majority)
2) Flaky test that may actually pass, so distinguishing what happens with
the image result is of limited utility (most of these are also text-only
tests; only a small subset even have an image result)
Thus, I think Fail and ImageOnlyFail would be more useful and understandable
categories than {TEXT, IMAGE, TEXT+IMAGE, TEXT IMAGE}. Fail would have the
semantic that a text failure is expected, and image result if any can either
pass or fail.
I too would like to see us remove TEXT+IMAGE. It's really confusing
to non-experts, and it doesn't scale as we introduce new kinds of
failures (like Audio). Do we really need TEXT+IMAGE+AUDIO,
TEXT+AUDIO, and IMAGE+AUDIO?
Adam
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 4:34 PM, Adam Barth aba...@webkit.org wrote: I too would like to see us remove TEXT+IMAGE. It's really confusing to non-experts, and it doesn't scale as we introduce new kinds of failures (like Audio). Do we really need TEXT+IMAGE+AUDIO, TEXT+AUDIO, and IMAGE+AUDIO? AUDIO tests can only produce audio output, not text or images, so, no. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 4:22 PM, Maciej Stachowiak m...@apple.com wrote:
On Jun 14, 2012, at 1:47 PM, Ryosuke Niwa rn...@webkit.org wrote:
On Thu, Jun 14, 2012 at 1:44 PM, Peter Kasting pkast...@chromium.org
wrote:
On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger epo...@chromium.org wrote:
Can someone please remind me why IMAGE+TEXT even exists?
Wouldn't it be simpler to just mark a test as follows?
IMAGE : allow image failure; go red if there is a text failure
TEXT: allow text failure; go red if there is an image failure
IMAGE TEXT: allow text and/or image failure
The distinction is that IMAGE TEXT will allow image, text, or both to
fail, thus making transitions among the three generate no events.
IMAGE+TEXT says specifically that we expect both to fail and that if one
starts passing, someone should do something. (For example, maybe someone
checks in a partial rebaseline where they miss the image expectations.)
Not to bike-shed on anything, but I think we should rename Text and Image to
TextOnly and ImageOnly. Every single person I know, including myself, had
never got the distinction between IMAGE TEXT and IMAGE+TEXT without someone
explaining it to him/her .
I think IMAGE+TEXT is not a very useful distinction from TEXT either. I
checked for uses of TEXT that is not IMAGE+TEXT in the Chromium
TextExpectations, and it seems that nearly all instances fall into one of
the two following categories:
1) text-only test, so IMAGE+TEXT would not have different semantics from
TEXT (the vast majority)
2) Flaky test that may actually pass, so distinguishing what happens with
the image result is of limited utility (most of these are also text-only
tests; only a small subset even have an image result)
Thus, I think Fail and ImageOnlyFail would be more useful and understandable
categories than {TEXT, IMAGE, TEXT+IMAGE, TEXT IMAGE}. Fail would have the
semantic that a text failure is expected, and image result if any can either
pass or fail.
This is perhaps true, but if it's okay I would like to treat that
feature request separately from the other syntactic changes we've been
discussing. So far the rest of the changes have not really implied any
changes to how we actually track which changes fail and how (note that
FAIL is different and we've fixed that separately from these changes
as well).
-- Dirk
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 4:34 PM, Adam Barth aba...@webkit.org wrote:
On Thu, Jun 14, 2012 at 4:22 PM, Maciej Stachowiak m...@apple.com wrote:
On Jun 14, 2012, at 1:47 PM, Ryosuke Niwa rn...@webkit.org wrote:
On Thu, Jun 14, 2012 at 1:44 PM, Peter Kasting pkast...@chromium.org
wrote:
On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger epo...@chromium.org
wrote:
Can someone please remind me why IMAGE+TEXT even exists?
Wouldn't it be simpler to just mark a test as follows?
IMAGE : allow image failure; go red if there is a text failure
TEXT: allow text failure; go red if there is an image failure
IMAGE TEXT: allow text and/or image failure
The distinction is that IMAGE TEXT will allow image, text, or both to
fail, thus making transitions among the three generate no events.
IMAGE+TEXT says specifically that we expect both to fail and that if
one
starts passing, someone should do something. (For example, maybe
someone
checks in a partial rebaseline where they miss the image expectations.)
Not to bike-shed on anything, but I think we should rename Text and
Image to
TextOnly and ImageOnly. Every single person I know, including myself, had
never got the distinction between IMAGE TEXT and IMAGE+TEXT without
someone
explaining it to him/her .
I think IMAGE+TEXT is not a very useful distinction from TEXT either. I
checked for uses of TEXT that is not IMAGE+TEXT in the Chromium
TextExpectations, and it seems that nearly all instances fall into one of
the two following categories:
1) text-only test, so IMAGE+TEXT would not have different semantics from
TEXT (the vast majority)
2) Flaky test that may actually pass, so distinguishing what happens with
the image result is of limited utility (most of these are also text-only
tests; only a small subset even have an image result)
Thus, I think Fail and ImageOnlyFail would be more useful and
understandable
categories than {TEXT, IMAGE, TEXT+IMAGE, TEXT IMAGE}. Fail would have
the
semantic that a text failure is expected, and image result if any can
either
pass or fail.
I too would like to see us remove TEXT+IMAGE. It's really confusing
to non-experts, and it doesn't scale as we introduce new kinds of
failures (like Audio). Do we really need TEXT+IMAGE+AUDIO,
TEXT+AUDIO, and IMAGE+AUDIO?
+1 to that. Also, I can never remember whether it's IMAGE+TEXT or
TEXT+IMAGE (it's IMAGE+TEXT).
But I agree with Dirk that we should probably discuss about this on a
separate thread.
- Ryosuke
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] IMAGE+TEXT WAS: TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 4:34 PM, Dirk Pranke dpra...@chromium.org wrote:
On Thu, Jun 14, 2012 at 4:22 PM, Maciej Stachowiak m...@apple.com wrote:
On Jun 14, 2012, at 1:47 PM, Ryosuke Niwa rn...@webkit.org wrote:
On Thu, Jun 14, 2012 at 1:44 PM, Peter Kasting pkast...@chromium.org
wrote:
On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger epo...@chromium.org
wrote:
Can someone please remind me why IMAGE+TEXT even exists?
Wouldn't it be simpler to just mark a test as follows?
IMAGE : allow image failure; go red if there is a text failure
TEXT: allow text failure; go red if there is an image failure
IMAGE TEXT: allow text and/or image failure
The distinction is that IMAGE TEXT will allow image, text, or both to
fail, thus making transitions among the three generate no events.
IMAGE+TEXT says specifically that we expect both to fail and that if
one
starts passing, someone should do something. (For example, maybe
someone
checks in a partial rebaseline where they miss the image expectations.)
Not to bike-shed on anything, but I think we should rename Text and
Image to
TextOnly and ImageOnly. Every single person I know, including myself, had
never got the distinction between IMAGE TEXT and IMAGE+TEXT without
someone
explaining it to him/her .
I think IMAGE+TEXT is not a very useful distinction from TEXT either. I
checked for uses of TEXT that is not IMAGE+TEXT in the Chromium
TextExpectations, and it seems that nearly all instances fall into one of
the two following categories:
1) text-only test, so IMAGE+TEXT would not have different semantics from
TEXT (the vast majority)
2) Flaky test that may actually pass, so distinguishing what happens with
the image result is of limited utility (most of these are also text-only
tests; only a small subset even have an image result)
Thus, I think Fail and ImageOnlyFail would be more useful and
understandable
categories than {TEXT, IMAGE, TEXT+IMAGE, TEXT IMAGE}. Fail would have
the
semantic that a text failure is expected, and image result if any can
either
pass or fail.
This is perhaps true, but if it's okay I would like to treat that
feature request separately from the other syntactic changes we've been
discussing. So far the rest of the changes have not really implied any
changes to how we actually track which changes fail and how (note that
FAIL is different and we've fixed that separately from these changes
as well).
Lets have the separate bikeshed. While this is less precise, I agree that
Fail and ImageOnlyFail would capture the vast majority use-case and remove
a frequent source of confusion and error. The big downside of this approach
is that a text-only failure that also starts failing the pixel result make
genuinely indicate a new bug. I think that happens rarely enough that I'm
OK with it for the added simplicity.
A couple open questions:
-Does Fail also replace Audio? Seems reasonable to me.
-What about reftest failures where there is no text comparison? I'd be fine
with saying you can do Fail or ImageOnlyFail and they mean the same thing
here.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] IMAGE+TEXT WAS: TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 8:56 PM, Ojan Vafai o...@chromium.org wrote:
On Thu, Jun 14, 2012 at 4:34 PM, Dirk Pranke dpra...@chromium.org wrote:
On Thu, Jun 14, 2012 at 4:22 PM, Maciej Stachowiak m...@apple.com wrote:
On Jun 14, 2012, at 1:47 PM, Ryosuke Niwa rn...@webkit.org wrote:
On Thu, Jun 14, 2012 at 1:44 PM, Peter Kasting pkast...@chromium.org
wrote:
On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger epo...@chromium.org
wrote:
Can someone please remind me why IMAGE+TEXT even exists?
Wouldn't it be simpler to just mark a test as follows?
IMAGE : allow image failure; go red if there is a text failure
TEXT: allow text failure; go red if there is an image failure
IMAGE TEXT: allow text and/or image failure
The distinction is that IMAGE TEXT will allow image, text, or both to
fail, thus making transitions among the three generate no events.
IMAGE+TEXT says specifically that we expect both to fail and that if
one
starts passing, someone should do something. (For example, maybe
someone
checks in a partial rebaseline where they miss the image expectations.)
Not to bike-shed on anything, but I think we should rename Text and
Image to
TextOnly and ImageOnly. Every single person I know, including myself,
had
never got the distinction between IMAGE TEXT and IMAGE+TEXT without
someone
explaining it to him/her .
I think IMAGE+TEXT is not a very useful distinction from TEXT either. I
checked for uses of TEXT that is not IMAGE+TEXT in the Chromium
TextExpectations, and it seems that nearly all instances fall into one
of
the two following categories:
1) text-only test, so IMAGE+TEXT would not have different semantics from
TEXT (the vast majority)
2) Flaky test that may actually pass, so distinguishing what happens
with
the image result is of limited utility (most of these are also text-only
tests; only a small subset even have an image result)
Thus, I think Fail and ImageOnlyFail would be more useful and
understandable
categories than {TEXT, IMAGE, TEXT+IMAGE, TEXT IMAGE}. Fail would have
the
semantic that a text failure is expected, and image result if any can
either
pass or fail.
This is perhaps true, but if it's okay I would like to treat that
feature request separately from the other syntactic changes we've been
discussing. So far the rest of the changes have not really implied any
changes to how we actually track which changes fail and how (note that
FAIL is different and we've fixed that separately from these changes
as well).
Lets have the separate bikeshed. While this is less precise, I agree that
Fail and ImageOnlyFail would capture the vast majority use-case and remove a
frequent source of confusion and error. The big downside of this approach is
that a text-only failure that also starts failing the pixel result make
genuinely indicate a new bug. I think that happens rarely enough that I'm OK
with it for the added simplicity.
A couple open questions:
-Does Fail also replace Audio? Seems reasonable to me.
Yeah, audio tests can fail only in one way.
-What about reftest failures where there is no text comparison? I'd be fine
with saying you can do Fail or ImageOnlyFail and they mean the same thing
here.
Similarly, I'd say that we should just Fail here. Reftests can fail
only in one way.
In this view, ImageOnlyFail is a special case for pixel tests because
they're so fragile.
Adam
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] IMAGE+TEXT WAS: TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 9:00 PM, Adam Barth aba...@webkit.org wrote:
On Thu, Jun 14, 2012 at 8:56 PM, Ojan Vafai o...@chromium.org wrote:
On Thu, Jun 14, 2012 at 4:34 PM, Dirk Pranke dpra...@chromium.org
wrote:
On Thu, Jun 14, 2012 at 4:22 PM, Maciej Stachowiak m...@apple.com
wrote:
On Jun 14, 2012, at 1:47 PM, Ryosuke Niwa rn...@webkit.org wrote:
On Thu, Jun 14, 2012 at 1:44 PM, Peter Kasting pkast...@chromium.org
wrote:
On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger epo...@chromium.org
wrote:
Can someone please remind me why IMAGE+TEXT even exists?
Wouldn't it be simpler to just mark a test as follows?
IMAGE : allow image failure; go red if there is a text failure
TEXT: allow text failure; go red if there is an image failure
IMAGE TEXT: allow text and/or image failure
The distinction is that IMAGE TEXT will allow image, text, or both to
fail, thus making transitions among the three generate no events.
IMAGE+TEXT says specifically that we expect both to fail and that if
one
starts passing, someone should do something. (For example, maybe
someone
checks in a partial rebaseline where they miss the image
expectations.)
Not to bike-shed on anything, but I think we should rename Text and
Image to
TextOnly and ImageOnly. Every single person I know, including myself,
had
never got the distinction between IMAGE TEXT and IMAGE+TEXT without
someone
explaining it to him/her .
I think IMAGE+TEXT is not a very useful distinction from TEXT either.
I
checked for uses of TEXT that is not IMAGE+TEXT in the Chromium
TextExpectations, and it seems that nearly all instances fall into one
of
the two following categories:
1) text-only test, so IMAGE+TEXT would not have different semantics
from
TEXT (the vast majority)
2) Flaky test that may actually pass, so distinguishing what happens
with
the image result is of limited utility (most of these are also
text-only
tests; only a small subset even have an image result)
Thus, I think Fail and ImageOnlyFail would be more useful and
understandable
categories than {TEXT, IMAGE, TEXT+IMAGE, TEXT IMAGE}. Fail would have
the
semantic that a text failure is expected, and image result if any can
either
pass or fail.
This is perhaps true, but if it's okay I would like to treat that
feature request separately from the other syntactic changes we've been
discussing. So far the rest of the changes have not really implied any
changes to how we actually track which changes fail and how (note that
FAIL is different and we've fixed that separately from these changes
as well).
Lets have the separate bikeshed. While this is less precise, I agree that
Fail and ImageOnlyFail would capture the vast majority use-case and
remove a
frequent source of confusion and error. The big downside of this
approach is
that a text-only failure that also starts failing the pixel result make
genuinely indicate a new bug. I think that happens rarely enough that
I'm OK
with it for the added simplicity.
A couple open questions:
-Does Fail also replace Audio? Seems reasonable to me.
Yeah, audio tests can fail only in one way.
-What about reftest failures where there is no text comparison? I'd be
fine
with saying you can do Fail or ImageOnlyFail and they mean the same thing
here.
Similarly, I'd say that we should just Fail here. Reftests can fail
only in one way.
Seems like it will be a common error to mark a reftest failure as
ImageOnlyFail and then be confused why it's not working, no?
In this view, ImageOnlyFail is a special case for pixel tests because
they're so fragile.
Adam
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] IMAGE+TEXT WAS: TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 9:02 PM, Ojan Vafai o...@chromium.org wrote:
On Thu, Jun 14, 2012 at 9:00 PM, Adam Barth aba...@webkit.org wrote:
On Thu, Jun 14, 2012 at 8:56 PM, Ojan Vafai o...@chromium.org wrote:
On Thu, Jun 14, 2012 at 4:34 PM, Dirk Pranke dpra...@chromium.org
wrote:
On Thu, Jun 14, 2012 at 4:22 PM, Maciej Stachowiak m...@apple.com
wrote:
On Jun 14, 2012, at 1:47 PM, Ryosuke Niwa rn...@webkit.org wrote:
On Thu, Jun 14, 2012 at 1:44 PM, Peter Kasting
pkast...@chromium.org
wrote:
On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger epo...@chromium.org
wrote:
Can someone please remind me why IMAGE+TEXT even exists?
Wouldn't it be simpler to just mark a test as follows?
IMAGE : allow image failure; go red if there is a text failure
TEXT: allow text failure; go red if there is an image failure
IMAGE TEXT: allow text and/or image failure
The distinction is that IMAGE TEXT will allow image, text, or both
to
fail, thus making transitions among the three generate no events.
IMAGE+TEXT says specifically that we expect both to fail and that
if
one
starts passing, someone should do something. (For example, maybe
someone
checks in a partial rebaseline where they miss the image
expectations.)
Not to bike-shed on anything, but I think we should rename Text and
Image to
TextOnly and ImageOnly. Every single person I know, including myself,
had
never got the distinction between IMAGE TEXT and IMAGE+TEXT without
someone
explaining it to him/her .
I think IMAGE+TEXT is not a very useful distinction from TEXT either.
I
checked for uses of TEXT that is not IMAGE+TEXT in the Chromium
TextExpectations, and it seems that nearly all instances fall into
one
of
the two following categories:
1) text-only test, so IMAGE+TEXT would not have different semantics
from
TEXT (the vast majority)
2) Flaky test that may actually pass, so distinguishing what happens
with
the image result is of limited utility (most of these are also
text-only
tests; only a small subset even have an image result)
Thus, I think Fail and ImageOnlyFail would be more useful and
understandable
categories than {TEXT, IMAGE, TEXT+IMAGE, TEXT IMAGE}. Fail would
have
the
semantic that a text failure is expected, and image result if any can
either
pass or fail.
This is perhaps true, but if it's okay I would like to treat that
feature request separately from the other syntactic changes we've been
discussing. So far the rest of the changes have not really implied any
changes to how we actually track which changes fail and how (note that
FAIL is different and we've fixed that separately from these changes
as well).
Lets have the separate bikeshed. While this is less precise, I agree
that
Fail and ImageOnlyFail would capture the vast majority use-case and
remove a
frequent source of confusion and error. The big downside of this
approach is
that a text-only failure that also starts failing the pixel result make
genuinely indicate a new bug. I think that happens rarely enough that
I'm OK
with it for the added simplicity.
A couple open questions:
-Does Fail also replace Audio? Seems reasonable to me.
Yeah, audio tests can fail only in one way.
-What about reftest failures where there is no text comparison? I'd be
fine
with saying you can do Fail or ImageOnlyFail and they mean the same
thing
here.
Similarly, I'd say that we should just Fail here. Reftests can fail
only in one way.
Seems like it will be a common error to mark a reftest failure as
ImageOnlyFail and then be confused why it's not working, no?
Maybe that can be solved with another name, like PixelOnlyFailure.
Adam
In this view, ImageOnlyFail is a special case for pixel tests because
they're so fragile.
Adam
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] IMAGE+TEXT WAS: TestExpectations syntax changes, last call (for a while, at least) ...
On Jun 14, 2012, at 9:06 PM, Adam Barth aba...@webkit.org wrote: On Thu, Jun 14, 2012 at 9:02 PM, Ojan Vafai o...@chromium.org wrote: Seems like it will be a common error to mark a reftest failure as ImageOnlyFail and then be confused why it's not working, no? Maybe that can be solved with another name, like PixelOnlyFailure. That sounds good. We could also make it an error to apply PixelOnlyFailure (or what have you) to a text-only test, a reftest, or an audio test. Error in the sense that it would be reported as a failure, with an informative diagnostic saying it does not apply. Regards, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] IMAGE+TEXT WAS: TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 9:20 PM, Maciej Stachowiak m...@apple.com wrote: On Jun 14, 2012, at 9:06 PM, Adam Barth aba...@webkit.org wrote: On Thu, Jun 14, 2012 at 9:02 PM, Ojan Vafai o...@chromium.org wrote: Seems like it will be a common error to mark a reftest failure as ImageOnlyFail and then be confused why it's not working, no? Maybe that can be solved with another name, like PixelOnlyFailure. I'm OK with trying it this way. We can always have another strikebikeshed/strike fruitful discussion if it turns out to be a frequent cause of confusion in practice. Not sure one name is any more clear than the other. PixelOnlyFailure seems fine to me since others have expressed a preference in the past for Pixel over Image. That sounds good. We could also make it an error to apply PixelOnlyFailure (or what have you) to a text-only test, a reftest, or an audio test. Error in the sense that it would be reported as a failure, with an informative diagnostic saying it does not apply. We already have a mechanism for errors like this. They are reported when you run the tests or when you run with --lint-test-files. At least on the chromium bots, this runs as a separate step that turns red when when you cause a lint failure. That way errors get noticed and addressed quickly (the lint step takes ~2 seconds to run). Ojan ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] IMAGE+TEXT WAS: TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 10:37 PM, Ojan Vafai o...@chromium.org wrote: On Thu, Jun 14, 2012 at 9:20 PM, Maciej Stachowiak m...@apple.com wrote: On Jun 14, 2012, at 9:06 PM, Adam Barth aba...@webkit.org wrote: On Thu, Jun 14, 2012 at 9:02 PM, Ojan Vafai o...@chromium.org wrote: Seems like it will be a common error to mark a reftest failure as ImageOnlyFail and then be confused why it's not working, no? Maybe that can be solved with another name, like PixelOnlyFailure. I'm OK with trying it this way. We can always have another strikebikeshed/strike fruitful discussion if it turns out to be a frequent cause of confusion in practice. Not sure one name is any more clear than the other. PixelOnlyFailure seems fine to me since others have expressed a preference in the past for Pixel over Image. I can certainly see the advantages of the suggested scheme, but I would want to sleep on this one, and possibly do some more data mining to see if I can figure out exactly what percentage of Chromium's tests are pixel tests marked as TEXT only failures or reftests are either just IMAGE or just TEXT. (I'm sure Maciej is right and the percentage is low, but I'd like to know how low). Further, once Chromium moves to a world where failing baselines are checked in, the percentage is probably somewhere between zero and inconsequential :). I don't think PixelOnlyFailure is any better than ImageOnlyFailure. That sounds good. We could also make it an error to apply PixelOnlyFailure (or what have you) to a text-only test, a reftest, or an audio test. Error in the sense that it would be reported as a failure, with an informative diagnostic saying it does not apply. We already have a mechanism for errors like this. They are reported when you run the tests or when you run with --lint-test-files. At least on the chromium bots, this runs as a separate step that turns red when when you cause a lint failure. That way errors get noticed and addressed quickly (the lint step takes ~2 seconds to run). We also will catch these errors on upload through the style checker (which is essentially just running --lint-test-files). As a side note, I need to add the lint step to the b.w.o bots ... -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
