Re: Disabling a few mochitests this week, and upcoming changes next week to browser-chrome
On 08/26/2014 12:52 PM, Gavin Sharp wrote: Thanks for the update! Very helpful. For bug 1058695, it looks like bz/khuey are trying to rope in Nikhil to help - Nikhil, I don't know what the rest of your current workload looks like, but if you could help chase that bug down that would be very helpful. I should be able to fix it tomorrow. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Disabling a few mochitests this week, and upcoming changes next week to browser-chrome
I wanted to post an update. We are very close to achieving all green tests on browser-chrome (tracking in bug 1057512)- there are 3 bugs remaining: *Bug 525284* https://bugzilla.mozilla.org/show_bug.cgi?id=525284 - browser_bug400731.js is fragile, not always passing ** patch for review, should be resolved this week *Bug 963075* https://bugzilla.mozilla.org/show_bug.cgi?id=963075 - browser_pdfjs_[main|views].js leaks until shutdown when run as a standalone directory ** ttaubert found a root cause (*Bug 1058695* https://bugzilla.mozilla.org/show_bug.cgi?id=1058695 - Promises can keep a page alive and even prevent Firefox from shutting down) *Bug 1041594* https://bugzilla.mozilla.org/show_bug.cgi?id=1041594 - browser_mozAudioChannel_muted.js crashes when run as a directory instead of a full suite ** initial patch, would like to know priority of this as it requires a bit more work We are really close to green, can we ensure Bug 1058695 and bug 1041594 have owners and a reasonably normal priority so we can finish this project? If this is unrealistic, we could discuss temporarily disabling the tests until there is time to fix the issues. Thanks again for all the work so far on this. -jmaher On Tue, Aug 19, 2014 at 3:08 PM, Gavin Sharp ga...@gavinsharp.com wrote: On Tue, Aug 19, 2014 at 9:06 AM, Ed Morley emor...@mozilla.com wrote: Agreed that this case is possibly slightly different - however this change is merely exposing already flaky tests - and this change is very much overdue. If the tests mentioned are really that important, then could we please get people allocated to fixing them? We have allocated people to fixing them. These bugs were explicitly assigned in the 34.2 iteration: Bug 947574 Bug 1002439 Bug 1041537 and we've made some significant progress in tracking them down. But as you know, fixing these issues can be relatively difficult. Ultimately no one individual test should have the right to destroy the SnR for a suite - and if we don't have this policy as a rapid (and enforceable in practice) means to control rogue tests, then the only other hammer the sheriffs have is to hide entire suites. If we're reduced to interacting with each other with hammers, then we've failed. Firefox peers will not ignore needinfo or review requests. If you feel that they are, then you can escalate to me as module owner. I do not ignore requests, and am quite responsive to email and IRC pings. Gavin ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Disabling a few mochitests this week, and upcoming changes next week to browser-chrome
Thanks for the update! Very helpful. For bug 1058695, it looks like bz/khuey are trying to rope in Nikhil to help - Nikhil, I don't know what the rest of your current workload looks like, but if you could help chase that bug down that would be very helpful. For bug 1041594, it looks like we need to figure out why Andrea's debug patch logging didn't show up in the log. Perhaps you two could connect to sort that out and get him a good try-debugging setup? Gavin On Tue, Aug 26, 2014 at 12:40 PM, Joel Maher joel.ma...@gmail.com wrote: I wanted to post an update. We are very close to achieving all green tests on browser-chrome (tracking in bug 1057512)- there are 3 bugs remaining: *Bug 525284* https://bugzilla.mozilla.org/show_bug.cgi?id=525284 - browser_bug400731.js is fragile, not always passing ** patch for review, should be resolved this week *Bug 963075* https://bugzilla.mozilla.org/show_bug.cgi?id=963075 - browser_pdfjs_[main|views].js leaks until shutdown when run as a standalone directory ** ttaubert found a root cause (*Bug 1058695* https://bugzilla.mozilla.org/show_bug.cgi?id=1058695 - Promises can keep a page alive and even prevent Firefox from shutting down) *Bug 1041594* https://bugzilla.mozilla.org/show_bug.cgi?id=1041594 - browser_mozAudioChannel_muted.js crashes when run as a directory instead of a full suite ** initial patch, would like to know priority of this as it requires a bit more work We are really close to green, can we ensure Bug 1058695 and bug 1041594 have owners and a reasonably normal priority so we can finish this project? If this is unrealistic, we could discuss temporarily disabling the tests until there is time to fix the issues. Thanks again for all the work so far on this. -jmaher On Tue, Aug 19, 2014 at 3:08 PM, Gavin Sharp ga...@gavinsharp.com wrote: On Tue, Aug 19, 2014 at 9:06 AM, Ed Morley emor...@mozilla.com wrote: Agreed that this case is possibly slightly different - however this change is merely exposing already flaky tests - and this change is very much overdue. If the tests mentioned are really that important, then could we please get people allocated to fixing them? We have allocated people to fixing them. These bugs were explicitly assigned in the 34.2 iteration: Bug 947574 Bug 1002439 Bug 1041537 and we've made some significant progress in tracking them down. But as you know, fixing these issues can be relatively difficult. Ultimately no one individual test should have the right to destroy the SnR for a suite - and if we don't have this policy as a rapid (and enforceable in practice) means to control rogue tests, then the only other hammer the sheriffs have is to hide entire suites. If we're reduced to interacting with each other with hammers, then we've failed. Firefox peers will not ignore needinfo or review requests. If you feel that they are, then you can escalate to me as module owner. I do not ignore requests, and am quite responsive to email and IRC pings. Gavin ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Disabling a few mochitests this week, and upcoming changes next week to browser-chrome
It has been 3+ weeks since Vaibhav and I found the remaining issues with --run-by-dir (https://bugzilla.mozilla.org/show_bug.cgi?id=992911) for browser-chrome. Since then a few issues have been fixed, and many have been ignored. Going with our test disabling policy (https://wiki.mozilla.org/Sheriffing/Test_Disabling_Policy), I am going to get ready and disable tests on Friday. This weekend I can sort out any remaining issues (there are always new issues that show up as the product changes) and turn on --run-by-dir next week. You can see all the bugs that are blocking bug 992911 (https://bugzilla.mozilla.org/showdependencytree.cgi?id=992911hide_resolved=1) Happy hacking, -jmaher ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Disabling a few mochitests this week, and upcoming changes next week to browser-chrome
Going with our test disabling policy (https://wiki.mozilla.org/Sheriffing/Test_Disabling_Policy), I am going to get ready and disable tests on Friday. A few issues here: - This particular case (we're making broad changes to how the tests run that causes many new failures) was not what that policy was meant to cover. We need some leeway to handle this situation differently. - This policy should probably clarify that any test disabling patches still require module owner/peer review. You can see all the bugs that are blocking bug 992911 (https://bugzilla.mozilla.org/showdependencytree.cgi?id=992911hide_resolved=1) We cannot disable all of the tests in that list wholesale. At least bug 1041569, bug 1041583, bug 1017187, bug 1001820, and bug 963075 need alternate solutions (i.e. more targeted disabling or fixes). Gavin On Tue, Aug 19, 2014 at 7:07 AM, jmaher joel.ma...@gmail.com wrote: It has been 3+ weeks since Vaibhav and I found the remaining issues with --run-by-dir (https://bugzilla.mozilla.org/show_bug.cgi?id=992911) for browser-chrome. Since then a few issues have been fixed, and many have been ignored. Going with our test disabling policy (https://wiki.mozilla.org/Sheriffing/Test_Disabling_Policy), I am going to get ready and disable tests on Friday. This weekend I can sort out any remaining issues (there are always new issues that show up as the product changes) and turn on --run-by-dir next week. You can see all the bugs that are blocking bug 992911 (https://bugzilla.mozilla.org/showdependencytree.cgi?id=992911hide_resolved=1) Happy hacking, -jmaher ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Disabling a few mochitests this week, and upcoming changes next week to browser-chrome
On Tuesday, August 19, 2014 11:46:08 AM UTC-4, Gavin Sharp wrote: Going with our test disabling policy (https://wiki.mozilla.org/Sheriffing/Test_Disabling_Policy), I am going to get ready and disable tests on Friday. A few issues here: - This particular case (we're making broad changes to how the tests run that causes many new failures) was not what that policy was meant to cover. We need some leeway to handle this situation differently. - This policy should probably clarify that any test disabling patches still require module owner/peer review. You can see all the bugs that are blocking bug 992911 (https://bugzilla.mozilla.org/showdependencytree.cgi?id=992911hide_resolved=1) We cannot disable all of the tests in that list wholesale. At least bug 1041569, bug 1041583, bug 1017187, bug 1001820, and bug 963075 Bug 1041569 - has been needinfo'd for 4 weeks, it is only disabling this on win debug, I don't know how to reduce that anymore, especially if nobody wants to look at it. bug 1041583 - no activity in 4+ week as well. We are only disabling it on windows debug bug 1017187 - no reply in 4+ weeks, this is disabling the test on all debug branches. bug 1001820 - idle for 3 weeks until last week and it was recommended to disable the test (in fact it appears to be disabled) bug 963075 - no activity for 4 weeks, we need to disable 2 pdf tests on opt builds only. All in all we are changing 12 tests in the manifests. It is obvious based on the lack of response in the bugs that fixing these bugs are not a priority, changing the way we run these tests will only give us more reliable results and less churn when we need to adjust chunking or jobs. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Disabling a few mochitests this week, and upcoming changes next week to browser-chrome
On 19/08/2014 16:46, Gavin Sharp wrote: A few issues here: - This particular case (we're making broad changes to how the tests run that causes many new failures) was not what that policy was meant to cover. We need some leeway to handle this situation differently. Agreed that this case is possibly slightly different - however this change is merely exposing already flaky tests - and this change is very much overdue. If the tests mentioned are really that important, then could we please get people allocated to fixing them? You can't exactly say that Joel hasn't given sufficient notification to the lists yet we've had virtually no interested in resolving the issues. - This policy should probably clarify that any test disabling patches still require module owner/peer review. The policy does state: In the rare case we are disabling the majority of the tests (either at once or slowly over time) for a given feature, we need to get the module owner to sign off on the current state of the tests. However it intentionally doesn't state it for smaller cases (I'm not including the above in this). Ultimately disabling a test is a last resort, and if done according to the policy, will have been performed after: In the case we go another 2 days with no response from a module owner, we will disable the test. ...in which case either the module owner will ignore the review request (which makes the policy pointless, since the whole idea was to set expectations and consequences that would actually be enforced), give it r- (which isn't acceptable, given they've not acted on the bug in a reasonable timeframe), or give it r+ (in which case, great, but this is the state we should be in anyway). Ultimately no one individual test should have the right to destroy the SnR for a suite - and if we don't have this policy as a rapid (and enforceable in practice) means to control rogue tests, then the only other hammer the sheriffs have is to hide entire suites. Kind regards, Ed ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Disabling a few mochitests this week, and upcoming changes next week to browser-chrome
(I'm trying to move this thread to firefox-dev, please send followups there.) Bug 1041569 - has been needinfo'd for 4 weeks, it is only disabling this on win debug, I don't know how to reduce that anymore, especially if nobody wants to look at it. Not what I see - a needinfo request was responded to on July 25th, and then there was no further activity (and no pending requests). Similar stories for the other bugs. Driving these bugs to completion is a shared responsibility - I am happy to help, but if this is important to you then you have some responsibility for communicating that importance and following up directly when you're not getting the results you need. That means you sometimes need to pester people, and need to escalate to a module owner (per the policy you linked to). There was no followup or escalation in those bugs. Gavin On Tue, Aug 19, 2014 at 9:00 AM, jmaher joel.ma...@gmail.com wrote: On Tuesday, August 19, 2014 11:46:08 AM UTC-4, Gavin Sharp wrote: Going with our test disabling policy (https://wiki.mozilla.org/Sheriffing/Test_Disabling_Policy), I am going to get ready and disable tests on Friday. A few issues here: - This particular case (we're making broad changes to how the tests run that causes many new failures) was not what that policy was meant to cover. We need some leeway to handle this situation differently. - This policy should probably clarify that any test disabling patches still require module owner/peer review. You can see all the bugs that are blocking bug 992911 (https://bugzilla.mozilla.org/showdependencytree.cgi?id=992911hide_resolved=1) We cannot disable all of the tests in that list wholesale. At least bug 1041569, bug 1041583, bug 1017187, bug 1001820, and bug 963075 Bug 1041569 - has been needinfo'd for 4 weeks, it is only disabling this on win debug, I don't know how to reduce that anymore, especially if nobody wants to look at it. bug 1041583 - no activity in 4+ week as well. We are only disabling it on windows debug bug 1017187 - no reply in 4+ weeks, this is disabling the test on all debug branches. bug 1001820 - idle for 3 weeks until last week and it was recommended to disable the test (in fact it appears to be disabled) bug 963075 - no activity for 4 weeks, we need to disable 2 pdf tests on opt builds only. All in all we are changing 12 tests in the manifests. It is obvious based on the lack of response in the bugs that fixing these bugs are not a priority, changing the way we run these tests will only give us more reliable results and less churn when we need to adjust chunking or jobs. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Disabling a few mochitests this week, and upcoming changes next week to browser-chrome
On Tue, Aug 19, 2014 at 9:06 AM, Ed Morley emor...@mozilla.com wrote: Agreed that this case is possibly slightly different - however this change is merely exposing already flaky tests - and this change is very much overdue. If the tests mentioned are really that important, then could we please get people allocated to fixing them? We have allocated people to fixing them. These bugs were explicitly assigned in the 34.2 iteration: Bug 947574 Bug 1002439 Bug 1041537 and we've made some significant progress in tracking them down. But as you know, fixing these issues can be relatively difficult. Ultimately no one individual test should have the right to destroy the SnR for a suite - and if we don't have this policy as a rapid (and enforceable in practice) means to control rogue tests, then the only other hammer the sheriffs have is to hide entire suites. If we're reduced to interacting with each other with hammers, then we've failed. Firefox peers will not ignore needinfo or review requests. If you feel that they are, then you can escalate to me as module owner. I do not ignore requests, and am quite responsive to email and IRC pings. Gavin ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
