Re: [chromium-dev] WebKit Gardeners 3 rebaseline.py -w
I started working on upstreaming the rebaseline tool, which is part of the plan for upstreaming layout tests in Q1. On Sat, Jan 9, 2010 at 2:53 PM, Eric Seidel esei...@chromium.org wrote: Once we have this sort of tool for webkit.org then those of us who work on webkit.org could just add the new baselines when we commit the patches in the first place. :) On Fri, Jan 8, 2010 at 5:38 PM, Julie Parent jpar...@chromium.org wrote: Yeah, me too. This is what tends to lead to me spending the day after my gardening rotation doing clean up. Maybe if we had 2 people gardening at the same time they could do this real time, but on a normal day, I think it is too much for one person. This tool is awesome though! Julie On Fri, Jan 8, 2010 at 4:54 PM, Jeremy Orlow jor...@chromium.org wrote: Same here. On Fri, Jan 8, 2010 at 4:51 PM, Drew Wilson atwil...@chromium.org wrote: Do you find that you have time to figure out if rebaselining a test is the right thing to do while you're actively gardening? Maybe I just work too slowly, but I often find that if I'm trying to rebaseline on the fly, it requires that I do at least *some* investigation of the test failure to make sure I'm not rebaselining in an error (or rebaselining a test that is merely flaky) which slows me down enough that I fall behind and inevitably am crushed by the WebKit juggernaut. -atw On Fri, Jan 8, 2010 at 4:33 PM, Dimitri Glazkov dglaz...@chromium.org wrote: Before heading out for the weekend, I just want to mention this: rebaseline tool really, really rocks. And yesterday I discovered an option that I, to my shame, hadn't seen before: -w. This option pulls baselines from the canary. It's like getting test expectations from the future! In other words, there are no more excuses for you, dear WebKit gardeners, to commit those ghastly BUG_SOMENAME entries in test_expectations. The workflow is mind-numbingly simple: 1) Identify tests that need rebaselining prior to rolling 2) Add these tests to test_expectations.txt as if you were to commit them -- except add a REBASELINE flag next to BUG_SOMENAME 3) Run rebaseline -w 4) Make sure that the tool ran and removed these entries from test_expectations.txt 5) Create CL -- you will notice how new expected result files are conveniently added for you. 6) Enjoy regression-free WebKit roll. Big thanks to Victor Wang for this amazing instrument of everlasting harmony. Send him your accolades. Or cash. If you find bugs/quirks, fix them. :DG P.S. As part of gardener/sheriff process overhaul, I will be adding these and other helpful tips/hints to our gardening doc. -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
Re: [chromium-dev] WebKit Gardeners 3 rebaseline.py -w
Once we have this sort of tool for webkit.org then those of us who work on webkit.org could just add the new baselines when we commit the patches in the first place. :) On Fri, Jan 8, 2010 at 5:38 PM, Julie Parent jpar...@chromium.org wrote: Yeah, me too. This is what tends to lead to me spending the day after my gardening rotation doing clean up. Maybe if we had 2 people gardening at the same time they could do this real time, but on a normal day, I think it is too much for one person. This tool is awesome though! Julie On Fri, Jan 8, 2010 at 4:54 PM, Jeremy Orlow jor...@chromium.org wrote: Same here. On Fri, Jan 8, 2010 at 4:51 PM, Drew Wilson atwil...@chromium.org wrote: Do you find that you have time to figure out if rebaselining a test is the right thing to do while you're actively gardening? Maybe I just work too slowly, but I often find that if I'm trying to rebaseline on the fly, it requires that I do at least *some* investigation of the test failure to make sure I'm not rebaselining in an error (or rebaselining a test that is merely flaky) which slows me down enough that I fall behind and inevitably am crushed by the WebKit juggernaut. -atw On Fri, Jan 8, 2010 at 4:33 PM, Dimitri Glazkov dglaz...@chromium.org wrote: Before heading out for the weekend, I just want to mention this: rebaseline tool really, really rocks. And yesterday I discovered an option that I, to my shame, hadn't seen before: -w. This option pulls baselines from the canary. It's like getting test expectations from the future! In other words, there are no more excuses for you, dear WebKit gardeners, to commit those ghastly BUG_SOMENAME entries in test_expectations. The workflow is mind-numbingly simple: 1) Identify tests that need rebaselining prior to rolling 2) Add these tests to test_expectations.txt as if you were to commit them -- except add a REBASELINE flag next to BUG_SOMENAME 3) Run rebaseline -w 4) Make sure that the tool ran and removed these entries from test_expectations.txt 5) Create CL -- you will notice how new expected result files are conveniently added for you. 6) Enjoy regression-free WebKit roll. Big thanks to Victor Wang for this amazing instrument of everlasting harmony. Send him your accolades. Or cash. If you find bugs/quirks, fix them. :DG P.S. As part of gardener/sheriff process overhaul, I will be adding these and other helpful tips/hints to our gardening doc. -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
[chromium-dev] WebKit Gardeners 3 rebaseline.py -w
Before heading out for the weekend, I just want to mention this: rebaseline tool really, really rocks. And yesterday I discovered an option that I, to my shame, hadn't seen before: -w. This option pulls baselines from the canary. It's like getting test expectations from the future! In other words, there are no more excuses for you, dear WebKit gardeners, to commit those ghastly BUG_SOMENAME entries in test_expectations. The workflow is mind-numbingly simple: 1) Identify tests that need rebaselining prior to rolling 2) Add these tests to test_expectations.txt as if you were to commit them -- except add a REBASELINE flag next to BUG_SOMENAME 3) Run rebaseline -w 4) Make sure that the tool ran and removed these entries from test_expectations.txt 5) Create CL -- you will notice how new expected result files are conveniently added for you. 6) Enjoy regression-free WebKit roll. Big thanks to Victor Wang for this amazing instrument of everlasting harmony. Send him your accolades. Or cash. If you find bugs/quirks, fix them. :DG P.S. As part of gardener/sheriff process overhaul, I will be adding these and other helpful tips/hints to our gardening doc. -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
Re: [chromium-dev] WebKit Gardeners 3 rebaseline.py -w
Do you find that you have time to figure out if rebaselining a test is the right thing to do while you're actively gardening? Maybe I just work too slowly, but I often find that if I'm trying to rebaseline on the fly, it requires that I do at least *some* investigation of the test failure to make sure I'm not rebaselining in an error (or rebaselining a test that is merely flaky) which slows me down enough that I fall behind and inevitably am crushed by the WebKit juggernaut. -atw On Fri, Jan 8, 2010 at 4:33 PM, Dimitri Glazkov dglaz...@chromium.orgwrote: Before heading out for the weekend, I just want to mention this: rebaseline tool really, really rocks. And yesterday I discovered an option that I, to my shame, hadn't seen before: -w. This option pulls baselines from the canary. It's like getting test expectations from the future! In other words, there are no more excuses for you, dear WebKit gardeners, to commit those ghastly BUG_SOMENAME entries in test_expectations. The workflow is mind-numbingly simple: 1) Identify tests that need rebaselining prior to rolling 2) Add these tests to test_expectations.txt as if you were to commit them -- except add a REBASELINE flag next to BUG_SOMENAME 3) Run rebaseline -w 4) Make sure that the tool ran and removed these entries from test_expectations.txt 5) Create CL -- you will notice how new expected result files are conveniently added for you. 6) Enjoy regression-free WebKit roll. Big thanks to Victor Wang for this amazing instrument of everlasting harmony. Send him your accolades. Or cash. If you find bugs/quirks, fix them. :DG P.S. As part of gardener/sheriff process overhaul, I will be adding these and other helpful tips/hints to our gardening doc. -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
Re: [chromium-dev] WebKit Gardeners 3 rebaseline.py -w
Same here. On Fri, Jan 8, 2010 at 4:51 PM, Drew Wilson atwil...@chromium.org wrote: Do you find that you have time to figure out if rebaselining a test is the right thing to do while you're actively gardening? Maybe I just work too slowly, but I often find that if I'm trying to rebaseline on the fly, it requires that I do at least *some* investigation of the test failure to make sure I'm not rebaselining in an error (or rebaselining a test that is merely flaky) which slows me down enough that I fall behind and inevitably am crushed by the WebKit juggernaut. -atw On Fri, Jan 8, 2010 at 4:33 PM, Dimitri Glazkov dglaz...@chromium.orgwrote: Before heading out for the weekend, I just want to mention this: rebaseline tool really, really rocks. And yesterday I discovered an option that I, to my shame, hadn't seen before: -w. This option pulls baselines from the canary. It's like getting test expectations from the future! In other words, there are no more excuses for you, dear WebKit gardeners, to commit those ghastly BUG_SOMENAME entries in test_expectations. The workflow is mind-numbingly simple: 1) Identify tests that need rebaselining prior to rolling 2) Add these tests to test_expectations.txt as if you were to commit them -- except add a REBASELINE flag next to BUG_SOMENAME 3) Run rebaseline -w 4) Make sure that the tool ran and removed these entries from test_expectations.txt 5) Create CL -- you will notice how new expected result files are conveniently added for you. 6) Enjoy regression-free WebKit roll. Big thanks to Victor Wang for this amazing instrument of everlasting harmony. Send him your accolades. Or cash. If you find bugs/quirks, fix them. :DG P.S. As part of gardener/sheriff process overhaul, I will be adding these and other helpful tips/hints to our gardening doc. -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
Re: [chromium-dev] WebKit Gardeners 3 rebaseline.py -w
Yeah, me too. This is what tends to lead to me spending the day after my gardening rotation doing clean up. Maybe if we had 2 people gardening at the same time they could do this real time, but on a normal day, I think it is too much for one person. This tool is awesome though! Julie On Fri, Jan 8, 2010 at 4:54 PM, Jeremy Orlow jor...@chromium.org wrote: Same here. On Fri, Jan 8, 2010 at 4:51 PM, Drew Wilson atwil...@chromium.org wrote: Do you find that you have time to figure out if rebaselining a test is the right thing to do while you're actively gardening? Maybe I just work too slowly, but I often find that if I'm trying to rebaseline on the fly, it requires that I do at least *some* investigation of the test failure to make sure I'm not rebaselining in an error (or rebaselining a test that is merely flaky) which slows me down enough that I fall behind and inevitably am crushed by the WebKit juggernaut. -atw On Fri, Jan 8, 2010 at 4:33 PM, Dimitri Glazkov dglaz...@chromium.orgwrote: Before heading out for the weekend, I just want to mention this: rebaseline tool really, really rocks. And yesterday I discovered an option that I, to my shame, hadn't seen before: -w. This option pulls baselines from the canary. It's like getting test expectations from the future! In other words, there are no more excuses for you, dear WebKit gardeners, to commit those ghastly BUG_SOMENAME entries in test_expectations. The workflow is mind-numbingly simple: 1) Identify tests that need rebaselining prior to rolling 2) Add these tests to test_expectations.txt as if you were to commit them -- except add a REBASELINE flag next to BUG_SOMENAME 3) Run rebaseline -w 4) Make sure that the tool ran and removed these entries from test_expectations.txt 5) Create CL -- you will notice how new expected result files are conveniently added for you. 6) Enjoy regression-free WebKit roll. Big thanks to Victor Wang for this amazing instrument of everlasting harmony. Send him your accolades. Or cash. If you find bugs/quirks, fix them. :DG P.S. As part of gardener/sheriff process overhaul, I will be adding these and other helpful tips/hints to our gardening doc. -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
