Re: [webkit-dev] PSA: run-webkit-tests now generates pixel results in retry
On Fri, Mar 22, 2013 at 3:24 PM, Ryosuke Niwa rn...@webkit.org wrote: Hi all, I've made a change to NRWT in http://trac.webkit.org/changeset/146657 so that it always generate pixel results in retries even when pixel tests are disabled by default (on all but Chromium ports). We now enable pixel tests while we're retrying tests and generate -expected.png results. This change has two important implications: - We can now grab -expected.png results from build.webkit.org builders for those tests that have resulted in non-image-only failures. - Chromium and Mac EWS bots will now upload ZIP archives with those images (recall http://trac.webkit.org/changeset/146443) This will allow us to rebaseline BOTH test and image expected results PRIOR to landing patches. Now that layout-test-results are being attached to their related bugs, I wonder if there is a way to delete these attachments (in BZ) so as not to choke disk space on the BZ server. Otherwise, I have a feeling disk usage will increase rapidly. For example, after one has reviewed and extracted the useful results from the attachments, it would be useful to delete (and not just make obsolete) those (large) attachments. I also notice that when an EWS has multiple fails on the same patch, that multiple layout-test-results get attached. It might also be useful to have a cron job on the BZ server go through and delete obsoleted layout-test-result attachments (in case it is not straightforward to permit attachment deletion). At least, then the dev could obsolete result attachments after extracting the useful data and the cron job could do the rest of the work of deleting the attachments. ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] PSA: run-webkit-tests now generates pixel results in retry
On Wed, Mar 27, 2013 at 9:16 AM, Glenn Adams gl...@skynav.com wrote: On Fri, Mar 22, 2013 at 3:24 PM, Ryosuke Niwa rn...@webkit.org wrote: Hi all, I've made a change to NRWT in http://trac.webkit.org/changeset/146657 so that it always generate pixel results in retries even when pixel tests are disabled by default (on all but Chromium ports). We now enable pixel tests while we're retrying tests and generate -expected.png results. This change has two important implications: - We can now grab -expected.png results from build.webkit.orgbuilders for those tests that have resulted in non-image-only failures. - Chromium and Mac EWS bots will now upload ZIP archives with those images (recall http://trac.webkit.org/changeset/146443) This will allow us to rebaseline BOTH test and image expected results PRIOR to landing patches. Now that layout-test-results are being attached to their related bugs, I wonder if there is a way to delete these attachments (in BZ) so as not to choke disk space on the BZ server. Otherwise, I have a feeling disk usage will increase rapidly. For example, after one has reviewed and extracted the useful results from the attachments, it would be useful to delete (and not just make obsolete) those (large) attachments. I also notice that when an EWS has multiple fails on the same patch, that multiple layout-test-results get attached. It might also be useful to have a cron job on the BZ server go through and delete obsoleted layout-test-result attachments (in case it is not straightforward to permit attachment deletion). At least, then the dev could obsolete result attachments after extracting the useful data and the cron job could do the rest of the work of deleting the attachments. Perhaps adding a DeleteObsoleteLayoutResults keyword would make automating deletion a viable process. It could either be added by default when the first layout results are attached (letting the dev decide whether to remove the keyword and keeping obsolete layout results) or the dev could explicitly add the keyword. I would suggest the former (add it by default), forcing an explicit action by the dev to retain obsolete layout results. ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] PSA: run-webkit-tests now generates pixel results in retry
On Wed, Mar 27, 2013 at 11:46 AM, Glenn Adams gl...@skynav.com wrote: On Wed, Mar 27, 2013 at 9:16 AM, Glenn Adams gl...@skynav.com wrote: On Fri, Mar 22, 2013 at 3:24 PM, Ryosuke Niwa rn...@webkit.org wrote: Hi all, I've made a change to NRWT in http://trac.webkit.org/changeset/146657so that it always generate pixel results in retries even when pixel tests are disabled by default (on all but Chromium ports). We now enable pixel tests while we're retrying tests and generate -expected.png results. This change has two important implications: - We can now grab -expected.png results from build.webkit.orgbuilders for those tests that have resulted in non-image-only failures. - Chromium and Mac EWS bots will now upload ZIP archives with those images (recall http://trac.webkit.org/changeset/146443) This will allow us to rebaseline BOTH test and image expected results PRIOR to landing patches. Now that layout-test-results are being attached to their related bugs, I wonder if there is a way to delete these attachments (in BZ) so as not to choke disk space on the BZ server. Otherwise, I have a feeling disk usage will increase rapidly. For example, after one has reviewed and extracted the useful results from the attachments, it would be useful to delete (and not just make obsolete) those (large) attachments. I also notice that when an EWS has multiple fails on the same patch, that multiple layout-test-results get attached. It might also be useful to have a cron job on the BZ server go through and delete obsoleted layout-test-result attachments (in case it is not straightforward to permit attachment deletion). At least, then the dev could obsolete result attachments after extracting the useful data and the cron job could do the rest of the work of deleting the attachments. Perhaps adding a DeleteObsoleteLayoutResults keyword would make automating deletion a viable process. It could either be added by default when the first layout results are attached (letting the dev decide whether to remove the keyword and keeping obsolete layout results) or the dev could explicitly add the keyword. I would suggest the former (add it by default), forcing an explicit action by the dev to retain obsolete layout results. I've filed a bug suggesting this enhancement: https://bugs.webkit.org/show_bug.cgi?id=113428 ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] PSA: run-webkit-tests now generates pixel results in retry
On Sat, Mar 23, 2013 at 10:25 PM, Robert Hogan li...@roberthogan.netwrote: This will allow us to rebaseline BOTH test and image expected results PRIOR to landing patches. You're welcome. Great stuff. Thanks for doing this! A question for the Qt/gtk/efl ports: do you plan to put some testers on the EWS queue? If not, I think you should shout if you do not want authors to add just the text part of new results and advise what they should do instead. Qt, GTK+, and EFL builders on build.webkit.org also generate image results on retries, meaning that as long as test unexpectedly fails, you can grab new -expected.png off of those builders. Given that, I'm not sure why you're asking this. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] PSA: run-webkit-tests now generates pixel results in retry
As far as I am informed we (Qt) are not running tests on EWS, due to limited resources. Therefore you would still not be able to receive the -expected.png files before landing your patch. But just right afterwards with the first bot failing. May be we need to consider changing our Qt-EWS? - Zeno On Sun, Mar 24, 2013 at 9:02 AM, Ryosuke Niwa rn...@webkit.org wrote: On Sat, Mar 23, 2013 at 10:25 PM, Robert Hogan li...@roberthogan.netwrote: This will allow us to rebaseline BOTH test and image expected results PRIOR to landing patches. You're welcome. Great stuff. Thanks for doing this! A question for the Qt/gtk/efl ports: do you plan to put some testers on the EWS queue? If not, I think you should shout if you do not want authors to add just the text part of new results and advise what they should do instead. Qt, GTK+, and EFL builders on build.webkit.org also generate image results on retries, meaning that as long as test unexpectedly fails, you can grab new -expected.png off of those builders. Given that, I'm not sure why you're asking this. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] PSA: run-webkit-tests now generates pixel results in retry
On Sun, Mar 24, 2013 at 2:06 AM, Zeno Albisser z...@webkit.org wrote: As far as I am informed we (Qt) are not running tests on EWS, due to limited resources. Therefore you would still not be able to receive the -expected.png files before landing your patch. But just right afterwards with the first bot failing. May be we need to consider changing our Qt-EWS? One thing we might want to consider doing is to add Rebaseline or NeedsTriaging keyword to TestExpectations and then have EWS just run tests that are marked with that keyword to generate text and image results for them. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] PSA: run-webkit-tests now generates pixel results in retry
This will allow us to rebaseline BOTH test and image expected results PRIOR to landing patches. You're welcome. Great stuff. Thanks for doing this! A question for the Qt/gtk/efl ports: do you plan to put some testers on the EWS queue? If not, I think you should shout if you do not want authors to add just the text part of new results and advise what they should do instead. ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] PSA: run-webkit-tests now generates pixel results in retry
Hi all, I've made a change to NRWT in http://trac.webkit.org/changeset/146657 so that it always generate pixel results in retries even when pixel tests are disabled by default (on all but Chromium ports). We now enable pixel tests while we're retrying tests and generate -expected.png results. This change has two important implications: - We can now grab -expected.png results from build.webkit.org builders for those tests that have resulted in non-image-only failures. - Chromium and Mac EWS bots will now upload ZIP archives with those images (recall http://trac.webkit.org/changeset/146443) This will allow us to rebaseline BOTH test and image expected results PRIOR to landing patches. You're welcome. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
