Re: [webkit-dev] Best practices for failing a flaky tests (was Re: Switching to new-run-webkit-tests)

2011-07-07 Thread Balazs Kelemen
On 07/06/2011 07:24 PM, Eric Seidel wrote: On Wed, Jul 6, 2011 at 10:18 AM, Xan Lopezx...@gnome.org wrote: On Wed, Jul 6, 2011 at 6:29 PM, Eric Seidele...@webkit.org wrote: NRWT uses both! It will read in all the port's Skipped files, covert them to SKIP text_expectations, and add them to

Re: [webkit-dev] Best practices for failing a flaky tests (was Re: Switching to new-run-webkit-tests)

2011-07-07 Thread Eric Seidel
I do not know the history as to why Chromium removed support for test_expectations cascading. Ideally we would have fewer test expectations, not more in the future. :) On Thu, Jul 7, 2011 at 3:16 AM, Balazs Kelemen kbal...@webkit.org wrote: On 07/06/2011 07:24 PM, Eric Seidel wrote: On Wed,

Re: [webkit-dev] Best practices for failing a flaky tests (was Re: Switching to new-run-webkit-tests)

2011-07-07 Thread Tony Chang
One difference with the chromium port is that we try to use a single test_expectations.txt that covers all platforms and OS versions (win xp, vista, 7, mac leopard, snow leopard, linux 32, 64, GPU vs CPU, Debug vs Release). The tokens to the left of the test name specify what configuration the

Re: [webkit-dev] Best practices for failing a flaky tests (was Re: Switching to new-run-webkit-tests)

2011-07-07 Thread Dirk Pranke
On Thu, Jul 7, 2011 at 10:27 AM, Tony Chang t...@chromium.org wrote: One difference with the chromium port is that we try to use a single test_expectations.txt that covers all platforms and OS versions (win xp, vista, 7, mac leopard, snow leopard, linux 32, 64, GPU vs CPU, Debug vs Release).  

Re: [webkit-dev] Best practices for failing a flaky tests (was Re: Switching to new-run-webkit-tests)

2011-07-07 Thread Maciej Stachowiak
On Jul 7, 2011, at 10:39 AM, Dirk Pranke wrote: On Thu, Jul 7, 2011 at 10:27 AM, Tony Chang t...@chromium.org wrote: One difference with the chromium port is that we try to use a single test_expectations.txt that covers all platforms and OS versions (win xp, vista, 7, mac leopard, snow

[webkit-dev] Best practices for failing a flaky tests (was Re: Switching to new-run-webkit-tests)

2011-07-06 Thread Adam Barth
I'm not sure we've quite figured that out yet. NRWT supports both Skipped lists and test_expectations.txt, which is a more expressive (but also more complex) version of Skipped lists. IMHO, we should wait for the dust to settle on the transition before changing our practices. Adam On Wed, Jul

Re: [webkit-dev] Best practices for failing a flaky tests (was Re: Switching to new-run-webkit-tests)

2011-07-06 Thread Adam Roben
On Jul 6, 2011, at 11:58 AM, Adam Barth wrote: I'm not sure we've quite figured that out yet. NRWT supports both Skipped lists and test_expectations.txt, which is a more expressive (but also more complex) version of Skipped lists. IMHO, we should wait for the dust to settle on the

Re: [webkit-dev] Best practices for failing a flaky tests (was Re: Switching to new-run-webkit-tests)

2011-07-06 Thread Ryosuke Niwa
On Wed, Jul 6, 2011 at 9:00 AM, Adam Roben aro...@apple.com wrote: OK. Then I have another question: What should I do to make the Leopard and SnowLeopard bots green, now that they have switched to NRWT? Looking at

Re: [webkit-dev] Best practices for failing a flaky tests (was Re: Switching to new-run-webkit-tests)

2011-07-06 Thread Adam Barth
On Wed, Jul 6, 2011 at 9:00 AM, Adam Roben aro...@apple.com wrote: On Jul 6, 2011, at 11:58 AM, Adam Barth wrote: I'm not sure we've quite figured that out yet.  NRWT supports both Skipped lists and test_expectations.txt, which is a more expressive (but also more complex) version of Skipped

Re: [webkit-dev] Best practices for failing a flaky tests (was Re: Switching to new-run-webkit-tests)

2011-07-06 Thread Xan Lopez
On Wed, Jul 6, 2011 at 6:29 PM, Eric Seidel e...@webkit.org wrote: NRWT uses both!  It will read in all the port's Skipped files, covert them to SKIP text_expectations, and add them to your test_expectations file.

Re: [webkit-dev] Best practices for failing a flaky tests (was Re: Switching to new-run-webkit-tests)

2011-07-06 Thread Eric Seidel
On Wed, Jul 6, 2011 at 10:18 AM, Xan Lopez x...@gnome.org wrote: On Wed, Jul 6, 2011 at 6:29 PM, Eric Seidel e...@webkit.org wrote: NRWT uses both!  It will read in all the port's Skipped files, covert them to SKIP text_expectations, and add them to your test_expectations file.