Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
Here are some examples of old and new format: BUGCR24182 SLOW MAC DEBUG : fast/css/large-list-of-rules-crash.html = PASS BUGCR24182 SLOW LINUX MAC DEBUG : fast/dom/Window/window-postmessage-clone-really-deep-array.html = PASS BUGCR24182 SLOW MAC DEBUG : fast/forms/select-set-length-with-mutation-remove.html = PASS crbug.com/24182 [Mac Debug] fast/css/large-list-of-rules-crash.html [Slow] crbug.com/24182 [Linux Mac Debug] fast/dom/Window/window-postmessage-clone-really-deep-array.html [Slow] crbug.com/24182 [Mac Debug] fast/dom/Window/window-postmessage-clone-really-deep-array.html [Slow] BUGWK87364 DEBUG : fast/dom/shadow/drop-event-for-input-in-shadow.html = TEXT PASS BUGWK87364 DEBUG : fast/dom/shadow/drop-event-in-shadow.html = TEXT PASS webkit.org/b/87364 [Debug] fast/dom/shadow/drop-event-for-input-in-shadow.html [Text Pass] webkit.org/b/87364 [Debug] fast/dom/shadow/drop-event-in-shadow.html [Text Pass] BUG_EAE SKIP : fast/repaint/reflection-repaint-test.html = PASS BUG_EAE SKIP : fast/repaint/transform-layout-repaint.html = PASS Bug(eae) fast/repaint/reflection-repaint-test.html [Skip] Bug(eae) fast/repaint/transform-layout-repaint.html [Skip] I've started to think that we might want to keep WontFix at the beginning of each line as supposed to placing them at the end of line as initially proposed: Beginning of line (current proposal with expectations): WONTFIX SKIP : media/media-captions.html = TIMEOUT BUGWK43459 WONTFIX : http/tests/appcache/origin-quota.html = MISSING TEXT IMAGE+TEXT media/media-captions.html [WontFix] webkit.org/b/43459 WontFix http/tests/appcache/origin-quota.html [WontFix Missing Text Image+Text] End of line: WONTFIX SKIP : media/media-captions.html = TIMEOUT BUGWK43459 WONTFIX : http/tests/appcache/origin-quota.html = MISSING TEXT IMAGE+TEXT WontFix media/media-captions.html WontFix webkit.org/b/43459 http/tests/appcache/origin-quota.html [Missing Text Image+Text] In fact, there are very few entries of WontFix with a bug number at least in Chromium port's test expectations file. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
Oops, some really bad typos. On Thu, Jun 14, 2012 at 3:10 AM, Ryosuke Niwa rn...@webkit.org wrote: Beginning of line (current proposal with expectations): End of line with expectations (current proposal) WONTFIX SKIP : media/media-captions.html = TIMEOUT BUGWK43459 WONTFIX : http/tests/appcache/origin-quota.html = MISSING TEXT IMAGE+TEXT media/media-captions.html [WontFix] webkit.org/b/43459 WontFix http/tests/appcache/origin-quota.html [WontFix Missing Text Image+Text] End of line: Beginning of line WONTFIX SKIP : media/media-captions.html = TIMEOUT BUGWK43459 WONTFIX : http/tests/appcache/origin-quota.html = MISSING TEXT IMAGE+TEXT WontFix media/media-captions.html WontFix webkit.org/b/43459 http/tests/appcache/origin-quota.html [Missing Text Image+Text] In fact, there are very few entries of WontFix with a bug number at least in Chromium port's test expectations file. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Wed, Jun 13, 2012 at 6:53 PM, Dirk Pranke dpra...@chromium.org wrote: * We'll probably rename IMAGE+TEXT to IMAGE_AND_TEXT. Can someone please remind me why IMAGE+TEXT even exists? Wouldn't it be simpler to just mark a test as follows? - IMAGE : allow image failure; go red if there is a text failure - TEXT: allow text failure; go red if there is an image failure - IMAGE TEXT: allow text and/or image failure ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger epo...@chromium.org wrote: Can someone please remind me why IMAGE+TEXT even exists? Wouldn't it be simpler to just mark a test as follows? - IMAGE : allow image failure; go red if there is a text failure - TEXT: allow text failure; go red if there is an image failure - IMAGE TEXT: allow text and/or image failure The distinction is that IMAGE TEXT will allow image, text, or both to fail, thus making transitions among the three generate no events. IMAGE+TEXT says specifically that we expect both to fail and that if one starts passing, someone should do something. (For example, maybe someone checks in a partial rebaseline where they miss the image expectations.) PK ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 1:44 PM, Peter Kasting pkast...@chromium.orgwrote: On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger epo...@chromium.org wrote: Can someone please remind me why IMAGE+TEXT even exists? Wouldn't it be simpler to just mark a test as follows? - IMAGE : allow image failure; go red if there is a text failure - TEXT: allow text failure; go red if there is an image failure - IMAGE TEXT: allow text and/or image failure The distinction is that IMAGE TEXT will allow image, text, or both to fail, thus making transitions among the three generate no events. IMAGE+TEXT says specifically that we expect both to fail and that if one starts passing, someone should do something. (For example, maybe someone checks in a partial rebaseline where they miss the image expectations.) Not to bike-shed on anything, but I think we should rename Text and Image to TextOnly and ImageOnly. Every single person I know, including myself, had never got the distinction between IMAGE TEXT and IMAGE+TEXT without someone explaining it to him/her . - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 4:44 PM, Peter Kasting pkast...@chromium.orgwrote: On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger epo...@chromium.org wrote: Can someone please remind me why IMAGE+TEXT even exists? Wouldn't it be simpler to just mark a test as follows? - IMAGE : allow image failure; go red if there is a text failure - TEXT: allow text failure; go red if there is an image failure - IMAGE TEXT: allow text and/or image failure The distinction is that IMAGE TEXT will allow image, text, or both to fail, thus making transitions among the three generate no events. IMAGE+TEXT says specifically that we expect both to fail and that if one starts passing, someone should do something. (For example, maybe someone checks in a partial rebaseline where they miss the image expectations.) Thanks for the explanation. That makes sense, although it seems to me that the problem of no-events-generated-by-changes-in-actual-images-while-IMAGE-failure-is-expected is about 100x worse for us. But that's not a reason to hide these particular transitions! :-) PK ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 1:44 PM, Peter Kasting pkast...@chromium.org wrote: On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger epo...@chromium.org wrote: Can someone please remind me why IMAGE+TEXT even exists? Wouldn't it be simpler to just mark a test as follows? IMAGE : allow image failure; go red if there is a text failure TEXT: allow text failure; go red if there is an image failure IMAGE TEXT: allow text and/or image failure The distinction is that IMAGE TEXT will allow image, text, or both to fail, thus making transitions among the three generate no events. IMAGE+TEXT says specifically that we expect both to fail and that if one starts passing, someone should do something. (For example, maybe someone checks in a partial rebaseline where they miss the image expectations.) Also, unlike the now-deleted FAIL keyword, IMAGE TEXT indicates that the test is expected to be flaky. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Jun 14, 2012, at 1:47 PM, Ryosuke Niwa rn...@webkit.org wrote:
On Thu, Jun 14, 2012 at 1:44 PM, Peter Kasting pkast...@chromium.org wrote:
On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger epo...@chromium.org wrote:
Can someone please remind me why IMAGE+TEXT even exists?
Wouldn't it be simpler to just mark a test as follows?
IMAGE : allow image failure; go red if there is a text failure
TEXT: allow text failure; go red if there is an image failure
IMAGE TEXT: allow text and/or image failure
The distinction is that IMAGE TEXT will allow image, text, or both to fail,
thus making transitions among the three generate no events. IMAGE+TEXT says
specifically that we expect both to fail and that if one starts passing,
someone should do something. (For example, maybe someone checks in a partial
rebaseline where they miss the image expectations.)
Not to bike-shed on anything, but I think we should rename Text and Image to
TextOnly and ImageOnly. Every single person I know, including myself, had
never got the distinction between IMAGE TEXT and IMAGE+TEXT without someone
explaining it to him/her .
I think IMAGE+TEXT is not a very useful distinction from TEXT either. I checked
for uses of TEXT that is not IMAGE+TEXT in the Chromium TextExpectations, and
it seems that nearly all instances fall into one of the two following
categories:
1) text-only test, so IMAGE+TEXT would not have different semantics from TEXT
(the vast majority)
2) Flaky test that may actually pass, so distinguishing what happens with the
image result is of limited utility (most of these are also text-only tests;
only a small subset even have an image result)
Thus, I think Fail and ImageOnlyFail would be more useful and understandable
categories than {TEXT, IMAGE, TEXT+IMAGE, TEXT IMAGE}. Fail would have the
semantic that a text failure is expected, and image result if any can either
pass or fail.
Regards,
Maciej
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 4:22 PM, Maciej Stachowiak m...@apple.com wrote:
On Jun 14, 2012, at 1:47 PM, Ryosuke Niwa rn...@webkit.org wrote:
On Thu, Jun 14, 2012 at 1:44 PM, Peter Kasting pkast...@chromium.org
wrote:
On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger epo...@chromium.org wrote:
Can someone please remind me why IMAGE+TEXT even exists?
Wouldn't it be simpler to just mark a test as follows?
IMAGE : allow image failure; go red if there is a text failure
TEXT: allow text failure; go red if there is an image failure
IMAGE TEXT: allow text and/or image failure
The distinction is that IMAGE TEXT will allow image, text, or both to
fail, thus making transitions among the three generate no events.
IMAGE+TEXT says specifically that we expect both to fail and that if one
starts passing, someone should do something. (For example, maybe someone
checks in a partial rebaseline where they miss the image expectations.)
Not to bike-shed on anything, but I think we should rename Text and Image to
TextOnly and ImageOnly. Every single person I know, including myself, had
never got the distinction between IMAGE TEXT and IMAGE+TEXT without someone
explaining it to him/her .
I think IMAGE+TEXT is not a very useful distinction from TEXT either. I
checked for uses of TEXT that is not IMAGE+TEXT in the Chromium
TextExpectations, and it seems that nearly all instances fall into one of
the two following categories:
1) text-only test, so IMAGE+TEXT would not have different semantics from
TEXT (the vast majority)
2) Flaky test that may actually pass, so distinguishing what happens with
the image result is of limited utility (most of these are also text-only
tests; only a small subset even have an image result)
Thus, I think Fail and ImageOnlyFail would be more useful and understandable
categories than {TEXT, IMAGE, TEXT+IMAGE, TEXT IMAGE}. Fail would have the
semantic that a text failure is expected, and image result if any can either
pass or fail.
I too would like to see us remove TEXT+IMAGE. It's really confusing
to non-experts, and it doesn't scale as we introduce new kinds of
failures (like Audio). Do we really need TEXT+IMAGE+AUDIO,
TEXT+AUDIO, and IMAGE+AUDIO?
Adam
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 4:34 PM, Adam Barth aba...@webkit.org wrote: I too would like to see us remove TEXT+IMAGE. It's really confusing to non-experts, and it doesn't scale as we introduce new kinds of failures (like Audio). Do we really need TEXT+IMAGE+AUDIO, TEXT+AUDIO, and IMAGE+AUDIO? AUDIO tests can only produce audio output, not text or images, so, no. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 4:22 PM, Maciej Stachowiak m...@apple.com wrote:
On Jun 14, 2012, at 1:47 PM, Ryosuke Niwa rn...@webkit.org wrote:
On Thu, Jun 14, 2012 at 1:44 PM, Peter Kasting pkast...@chromium.org
wrote:
On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger epo...@chromium.org wrote:
Can someone please remind me why IMAGE+TEXT even exists?
Wouldn't it be simpler to just mark a test as follows?
IMAGE : allow image failure; go red if there is a text failure
TEXT: allow text failure; go red if there is an image failure
IMAGE TEXT: allow text and/or image failure
The distinction is that IMAGE TEXT will allow image, text, or both to
fail, thus making transitions among the three generate no events.
IMAGE+TEXT says specifically that we expect both to fail and that if one
starts passing, someone should do something. (For example, maybe someone
checks in a partial rebaseline where they miss the image expectations.)
Not to bike-shed on anything, but I think we should rename Text and Image to
TextOnly and ImageOnly. Every single person I know, including myself, had
never got the distinction between IMAGE TEXT and IMAGE+TEXT without someone
explaining it to him/her .
I think IMAGE+TEXT is not a very useful distinction from TEXT either. I
checked for uses of TEXT that is not IMAGE+TEXT in the Chromium
TextExpectations, and it seems that nearly all instances fall into one of
the two following categories:
1) text-only test, so IMAGE+TEXT would not have different semantics from
TEXT (the vast majority)
2) Flaky test that may actually pass, so distinguishing what happens with
the image result is of limited utility (most of these are also text-only
tests; only a small subset even have an image result)
Thus, I think Fail and ImageOnlyFail would be more useful and understandable
categories than {TEXT, IMAGE, TEXT+IMAGE, TEXT IMAGE}. Fail would have the
semantic that a text failure is expected, and image result if any can either
pass or fail.
This is perhaps true, but if it's okay I would like to treat that
feature request separately from the other syntactic changes we've been
discussing. So far the rest of the changes have not really implied any
changes to how we actually track which changes fail and how (note that
FAIL is different and we've fixed that separately from these changes
as well).
-- Dirk
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 4:34 PM, Adam Barth aba...@webkit.org wrote:
On Thu, Jun 14, 2012 at 4:22 PM, Maciej Stachowiak m...@apple.com wrote:
On Jun 14, 2012, at 1:47 PM, Ryosuke Niwa rn...@webkit.org wrote:
On Thu, Jun 14, 2012 at 1:44 PM, Peter Kasting pkast...@chromium.org
wrote:
On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger epo...@chromium.org
wrote:
Can someone please remind me why IMAGE+TEXT even exists?
Wouldn't it be simpler to just mark a test as follows?
IMAGE : allow image failure; go red if there is a text failure
TEXT: allow text failure; go red if there is an image failure
IMAGE TEXT: allow text and/or image failure
The distinction is that IMAGE TEXT will allow image, text, or both to
fail, thus making transitions among the three generate no events.
IMAGE+TEXT says specifically that we expect both to fail and that if
one
starts passing, someone should do something. (For example, maybe
someone
checks in a partial rebaseline where they miss the image expectations.)
Not to bike-shed on anything, but I think we should rename Text and
Image to
TextOnly and ImageOnly. Every single person I know, including myself, had
never got the distinction between IMAGE TEXT and IMAGE+TEXT without
someone
explaining it to him/her .
I think IMAGE+TEXT is not a very useful distinction from TEXT either. I
checked for uses of TEXT that is not IMAGE+TEXT in the Chromium
TextExpectations, and it seems that nearly all instances fall into one of
the two following categories:
1) text-only test, so IMAGE+TEXT would not have different semantics from
TEXT (the vast majority)
2) Flaky test that may actually pass, so distinguishing what happens with
the image result is of limited utility (most of these are also text-only
tests; only a small subset even have an image result)
Thus, I think Fail and ImageOnlyFail would be more useful and
understandable
categories than {TEXT, IMAGE, TEXT+IMAGE, TEXT IMAGE}. Fail would have
the
semantic that a text failure is expected, and image result if any can
either
pass or fail.
I too would like to see us remove TEXT+IMAGE. It's really confusing
to non-experts, and it doesn't scale as we introduce new kinds of
failures (like Audio). Do we really need TEXT+IMAGE+AUDIO,
TEXT+AUDIO, and IMAGE+AUDIO?
+1 to that. Also, I can never remember whether it's IMAGE+TEXT or
TEXT+IMAGE (it's IMAGE+TEXT).
But I agree with Dirk that we should probably discuss about this on a
separate thread.
- Ryosuke
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
Hi all, Because I have infinite patience for bikeshedding, I thought I would send out Yet Another note on the proposed changes to the expectation syntax. Based on the last thread, I'm planning to change ORWT so that it will recognize the syntax in the TestExpectations files and treat any non-PASS entry as Skipped, so we will be able to switch over to TestExpectations files wholesale, so you might want to care about this ... I would also like to batch all of the changes up at once and then defer any changes for a month or more, just so the syntax isn't changing frequently and we have to retrain everyone. I'm of course open to changing things whenever we need to :). The current syntax is documented here: http://trac.webkit.org/wiki/TestExpectations In https://bugs.webkit.org/show_bug.cgi?id=86796 , we are proposing to change it to something like: webkit.org/12345 WIN MAC DEBUG \ animations/stop-animation-on-suspend.html \ CRASH TEXT PASS * the left hand side of the test now just contains the bug link and keywords that modify which configurations the line applies to. * You may use bug(dpranke) instead of a URL on the left hand side of the expression. * The REBASELINE, SLOW, SKIP, WONTFIX keywords will move to the right-hand side (note that REBASELINE doesn't ever actually get checked in). * we use \ (backslash) as a delimiter instead of : and = * The FAIL keyword is being removed as we speak. * We'll probably rename IMAGE+TEXT to IMAGE_AND_TEXT. * WONTFIX will have the same effect as SKIP but has a different meaning: we don't plan on passing the test at all; SKIP is intended to work around temporary bugs or problems. * As of now the keywords remain all UPPERCASE. I'm happy to change them to all lowercase or mixed case if there's a consensus that such a change is desired. Speak now if you really don't like these proposals, or hold your peace for at least a month ;) -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Wed, Jun 13, 2012 at 3:58 PM, Darin Adler da...@apple.com wrote: On Jun 13, 2012, at 3:53 PM, Dirk Pranke dpra...@chromium.org wrote: * we use \ (backslash) as a delimiter instead of : and = Seems worse to me. When I see a backslash I assume it’s a line continuation character or a C escape sequence. Gotta admit this one mystifies me too. I must have missed where someone suggested this in the last thread. * As of now the keywords remain all UPPERCASE. I'm happy to change them to all lowercase or mixed case if there's a consensus that such a change is desired. Given the last thread it seems clear there will not be consensus on any outcome of this particular question. PK ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Wed, Jun 13, 2012 at 4:12 PM, Benjamin Poulain benja...@webkit.orgwrote: On Wed, Jun 13, 2012 at 3:53 PM, Dirk Pranke dpra...@chromium.org wrote: webkit.org/12345 WIN MAC DEBUG \ animations/stop-animation-on-suspend.html \ CRASH TEXT PASS My bikeshedding: -In my opinion, the backslash is not any better than the old delimiter. It looks like we are escaping something. I liked better the proposal to have platform and results grouped instead of the delimiter. e.g: webkit.org/12345 (WIN MAC DEBUG) animations/stop-animation-on-suspend.html (CRASH TEXT PASS) -I prefer lowercase keywords, possibly with the first character uppercase (Crash, Text, Pass, Fail, Win, Mac, Debug). Do we use all uppercase keywords somewhere else in the project? So something like webkit.org/12345 [Win Mac Debug] animations/stop-animation-on-suspend.html [Crash Text Pass] ? I like that! - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Wed, Jun 13, 2012 at 3:58 PM, Darin Adler da...@apple.com wrote: On Jun 13, 2012, at 3:53 PM, Dirk Pranke dpra...@chromium.org wrote: * we use \ (backslash) as a delimiter instead of : and = Seems worse to me. When I see a backslash I assume it’s a line continuation character or a C escape sequence. On the other hand, neither the : nor the = meant anything to me either, and I suggested dropping the delimiter entirely, so I’m not sure my feedback will get listened to! That's a bit harsh. listened to does not necessarily mean will do what you ask :). Plenty of other people have said that they liked delimiters, so I think you just got outvoted. Are there other delimiters you might prefer to ':' and '=' ? Others suggestions have included bracketing the test name , and or uniformly using one of : , = - ; , or as Benjamin or Ryoskue have suggested, bracketing the keywords ... For the record, I didn't like bracketing the test name since brackets make it look optional. I could be open to bracketing the modifiers and keywords, since the modifiers are optional, and it would seem reasonable to make the absence of keywords equivalent to SKIP ... * As of now the keywords remain all UPPERCASE. I'm happy to change them to all lowercase or mixed case if there's a consensus that such a change is desired. I DON'T LIKE READING THINGS IN ALL UPPERCASE AND WOULD VERY MUCH LIKE THAT TO CHANGE. Would you prefer lowercase, lowercase_with_underscores, Initialcaps, MixedCase ... ? On Wed, Jun 13, 2012 at 4:12 PM, Benjamin Poulain benja...@webkit.org wrote: On Wed, Jun 13, 2012 at 3:53 PM, Dirk Pranke dpra...@chromium.org wrote: webkit.org/12345 WIN MAC DEBUG \ animations/stop-animation-on-suspend.html \ CRASH TEXT PASS My bikeshedding: -In my opinion, the backslash is not any better than the old delimiter. It looks like we are escaping something. I liked better the proposal to have platform and results grouped instead of the delimiter. e.g: webkit.org/12345 (WIN MAC DEBUG) animations/stop-animation-on-suspend.html (CRASH TEXT PASS) -I prefer lowercase keywords, possibly with the first character uppercase (Crash, Text, Pass, Fail, Win, Mac, Debug). Do we use all uppercase keywords somewhere else in the project? C++ macros are the only place I know of. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Wed, Jun 13, 2012 at 4:26 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Jun 13, 2012 at 4:12 PM, Benjamin Poulain benja...@webkit.org wrote: On Wed, Jun 13, 2012 at 3:53 PM, Dirk Pranke dpra...@chromium.org wrote: webkit.org/12345 WIN MAC DEBUG \ animations/stop-animation-on-suspend.html \ CRASH TEXT PASS My bikeshedding: -In my opinion, the backslash is not any better than the old delimiter. It looks like we are escaping something. I liked better the proposal to have platform and results grouped instead of the delimiter. e.g: webkit.org/12345 (WIN MAC DEBUG) animations/stop-animation-on-suspend.html (CRASH TEXT PASS) -I prefer lowercase keywords, possibly with the first character uppercase (Crash, Text, Pass, Fail, Win, Mac, Debug). Do we use all uppercase keywords somewhere else in the project? So something like webkit.org/12345 [Win Mac Debug] animations/stop-animation-on-suspend.html [Crash Text Pass] ? I like that! This seems quite readable to me as well. As mentioned above, I'd be inclined to make the absence of [] on the right hand side mean SKIP. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Wed, Jun 13, 2012 at 4:32 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, Jun 13, 2012 at 4:26 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Jun 13, 2012 at 4:12 PM, Benjamin Poulain benja...@webkit.org wrote: On Wed, Jun 13, 2012 at 3:53 PM, Dirk Pranke dpra...@chromium.org wrote: webkit.org/12345 WIN MAC DEBUG \ animations/stop-animation-on-suspend.html \ CRASH TEXT PASS My bikeshedding: -In my opinion, the backslash is not any better than the old delimiter. It looks like we are escaping something. I liked better the proposal to have platform and results grouped instead of the delimiter. e.g: webkit.org/12345 (WIN MAC DEBUG) animations/stop-animation-on-suspend.html (CRASH TEXT PASS) -I prefer lowercase keywords, possibly with the first character uppercase (Crash, Text, Pass, Fail, Win, Mac, Debug). Do we use all uppercase keywords somewhere else in the project? So something like webkit.org/12345 [Win Mac Debug] animations/stop-animation-on-suspend.html [Crash Text Pass] ? I like that! This seems quite readable to me as well. As mentioned above, I'd be inclined to make the absence of [] on the right hand side mean SKIP. That sounds very reasonable. In fact, I'd argue that it should be the only way to skip tests. It makes no sense to have any other test expectation when a test is skipped because the types of failures of skipped tests (when actually ran) will very likely change over time, and there is no way to track them. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Wed, Jun 13, 2012 at 4:38 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Jun 13, 2012 at 4:32 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, Jun 13, 2012 at 4:26 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Jun 13, 2012 at 4:12 PM, Benjamin Poulain benja...@webkit.org wrote: On Wed, Jun 13, 2012 at 3:53 PM, Dirk Pranke dpra...@chromium.org wrote: webkit.org/12345 WIN MAC DEBUG \ animations/stop-animation-on-suspend.html \ CRASH TEXT PASS My bikeshedding: -In my opinion, the backslash is not any better than the old delimiter. It looks like we are escaping something. I liked better the proposal to have platform and results grouped instead of the delimiter. e.g: webkit.org/12345 (WIN MAC DEBUG) animations/stop-animation-on-suspend.html (CRASH TEXT PASS) -I prefer lowercase keywords, possibly with the first character uppercase (Crash, Text, Pass, Fail, Win, Mac, Debug). Do we use all uppercase keywords somewhere else in the project? So something like webkit.org/12345 [Win Mac Debug] animations/stop-animation-on-suspend.html [Crash Text Pass] ? I like that! This seems quite readable to me as well. As mentioned above, I'd be inclined to make the absence of [] on the right hand side mean SKIP. That sounds very reasonable. In fact, I'd argue that it should be the only way to skip tests. It makes no sense to have any other test expectation when a test is skipped because the types of failures of skipped tests (when actually ran) will very likely change over time, and there is no way to track them. There's still a distinction between SKIP (temporary) and WONTFIX (permanent until some plan changes) that we should preserve, I think. It could be useful to indicate SKIP CRASH or SKIP TIMEOUT as well, but I agree that the type of failure will probably change over time. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Wed, Jun 13, 2012 at 4:32 PM, Dirk Pranke dpra...@chromium.org wrote: So something like webkit.org/12345 [Win Mac Debug] animations/stop-animation-on-suspend.html [Crash Text Pass] ? I like that! This seems quite readable to me as well. As mentioned above, I'd be inclined to make the absence of [] on the right hand side mean SKIP. I like that. Maybe TestExpectations is not the right name because of Skip? TestExceptions would be better? I don't care about the name of the file, this is just a random idea. Benjamin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
As long as we're considering TitleCase for the keywords, could we use it to keep all of them as single words? WontFix, SkipCrash, SkipTimeout --Tom On Jun 13, 2012, at 4:46 PM, Dirk Pranke wrote: There's still a distinction between SKIP (temporary) and WONTFIX (permanent until some plan changes) that we should preserve, I think. It could be useful to indicate SKIP CRASH or SKIP TIMEOUT as well, but I agree that the type of failure will probably change over time. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev On Jun 13, 2012 at 4:32 PM, Dirk Pranke dpra...@chromium.org wrote: So something like webkit.org/12345 [Win Mac Debug] animations/stop-animation-on-suspend.html [Crash Text Pass] ? I like that! This seems quite readable to me as well. As mentioned above, I'd be inclined to make the absence of [] on the right hand side mean SKIP. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
Skip and WontFix have a different purpose than Crash and timeout; the first two (like slow) tell you what to do with the test, and the latter tell you what you expect the test to result in. I don't want to combine them into single tokens because that would cause an explosion of tokens :). -- Dirk On Wed, Jun 13, 2012 at 4:55 PM, Tom Zakrajsek t...@codeaurora.org wrote: As long as we're considering TitleCase for the keywords, could we use it to keep all of them as single words? WontFix, SkipCrash, SkipTimeout --Tom On Jun 13, 2012, at 4:46 PM, Dirk Pranke wrote: There's still a distinction between SKIP (temporary) and WONTFIX (permanent until some plan changes) that we should preserve, I think. It could be useful to indicate SKIP CRASH or SKIP TIMEOUT as well, but I agree that the type of failure will probably change over time. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev On Jun 13, 2012 at 4:32 PM, Dirk Pranke dpra...@chromium.org wrote: So something like webkit.org/12345 [Win Mac Debug] animations/stop-animation-on-suspend.html [Crash Text Pass] ? I like that! This seems quite readable to me as well. As mentioned above, I'd be inclined to make the absence of [] on the right hand side mean SKIP. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Wed, Jun 13, 2012 at 4:55 PM, Tom Zakrajsek t...@codeaurora.org wrote: As long as we're considering TitleCase for the keywords, could we use it to keep all of them as single words? WontFix, SkipCrash, SkipTimeout For skipped tests, it doesn't make sense to have crash, timeout, etc... because we wouldn't know even if failure types changed from text to crash, or from timeout to image. It's just a pure noise as far as I'm concerned. On Jun 13, 2012, at 4:46 PM, Dirk Pranke wrote: There's still a distinction between SKIP (temporary) and WONTFIX (permanent until some plan changes) that we should preserve, I think. The way I see it, WontFix and NotImplemented, like bug URL, are orthogonal to other test expectations. They tell us why we have a given test expectation but doesn't define the expectation itself. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Wed, Jun 13, 2012 at 5:05 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Jun 13, 2012 at 4:55 PM, Tom Zakrajsek t...@codeaurora.org wrote: As long as we're considering TitleCase for the keywords, could we use it to keep all of them as single words? WontFix, SkipCrash, SkipTimeout For skipped tests, it doesn't make sense to have crash, timeout, etc... because we wouldn't know even if failure types changed from text to crash, or from timeout to image. It's just a pure noise as far as I'm concerned. Well, when you add the skip rule, you presumably know what happens when you run the test. Since you can actually run tests that are skipped using --force or --skipped=ignore/only (which I'm adding to NRWT as we speak), it's not quite noise, but I agree that it's pretty close. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Jun 13, 2012, at 3:58 PM, Darin Adler da...@apple.com wrote: On Jun 13, 2012, at 3:53 PM, Dirk Pranke dpra...@chromium.org wrote: * we use \ (backslash) as a delimiter instead of : and = Seems worse to me. When I see a backslash I assume it’s a line continuation character or a C escape sequence. On the other hand, neither the : nor the = meant anything to me either, and I suggested dropping the delimiter entirely, so I’m not sure my feedback will get listened to! * As of now the keywords remain all UPPERCASE. I'm happy to change them to all lowercase or mixed case if there's a consensus that such a change is desired. I DON'T LIKE READING THINGS IN ALL UPPERCASE AND WOULD VERY MUCH LIKE THAT TO CHANGE. I also hate all-caps keywords and would prefer mixed case or all-lowercase, but I'm willing to discuss that for a future round of changes if we can't get agreement now. Regards, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Wed, Jun 13, 2012 at 5:42 PM, Maciej Stachowiak m...@apple.com wrote: On Jun 13, 2012, at 3:58 PM, Darin Adler da...@apple.com wrote: On Jun 13, 2012, at 3:53 PM, Dirk Pranke dpra...@chromium.org wrote: * we use \ (backslash) as a delimiter instead of : and = Seems worse to me. When I see a backslash I assume it’s a line continuation character or a C escape sequence. On the other hand, neither the : nor the = meant anything to me either, and I suggested dropping the delimiter entirely, so I’m not sure my feedback will get listened to! * As of now the keywords remain all UPPERCASE. I'm happy to change them to all lowercase or mixed case if there's a consensus that such a change is desired. I DON'T LIKE READING THINGS IN ALL UPPERCASE AND WOULD VERY MUCH LIKE THAT TO CHANGE. I also hate all-caps keywords and would prefer mixed case or all-lowercase, but I'm willing to discuss that for a future round of changes if we can't get agreement now. I think most of the people who liked all-caps mostly wanted there to be *some* sort of delimiter. Anyone not okay with: webkit.org/12345 [Win Mac Debug] animations/stop-animation-on-suspend.html [Crash Text Pass] ? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Jun 13, 2012, at 4:26 PM, Ryosuke Niwa rn...@webkit.org wrote: So something like webkit.org/12345 [Win Mac Debug] animations/stop-animation-on-suspend.html [Crash Text Pass] ? I like that! Yes, looks good to me too. -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
