Update: Dan Duval and I sat down and wrote this task to help the generic case for these problems in future - https://phabricator.wikimedia.org/T101062
The good news: The caching issue has gone and Gather has gone green (yay!) https://integration.wikimedia.org/ci/view/Mobile/job/browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/ MobileFrontend cache problem has also gone away in the smoke tests but actually has surfaced a bug. The next build should go green thanks to https://gerrit.wikimedia.org/r/215257 so watch this space. Bad news: the mega builds are still failing (so 2 e-mails should be coming in rather than 4). Dan Duval and I are going to pair on this tomorrow and fix those up: https://phabricator.wikimedia.org/T101071 On Mon, Jun 1, 2015 at 10:11 AM, Jon Robson <jrob...@wikimedia.org> wrote: > Who would be suitable to debug these with and/or examine the caching on > beta labs? Whilst they fail like this they are useless to us (I've already > had a request to turn off the e-mail notifications). > > It seems Bryan had an idea on the related bug > https://phabricator.wikimedia.org/T99096 > > I am free today (and in the office) if anyone wants to pair and get this > fixed... > > On Thu, May 28, 2015 at 9:53 AM, Greg Grossmeier <g...@wikimedia.org> > wrote: > >> (let's use mailing lists instead of manually cc'ing people, please) >> >> Timo/Jon: Do you have tasks in phabricator that show what you've tried >> doing/debugging already? Would hate to re-do it all. >> >> Greg >> >> >> <quote name="Timo Tijhof" date="2015-05-28" time="11:57:05 +0200"> >> > I've seen this and similar issues on beta cluster for almost a year now >> and am reasonably certain that there is no bug in ResourceLoader related to >> this. We might uncover an indirect cause in MediaWiki core later, but the >> immediate cause is most likely in Beta cluster. This because I've observed >> the relevant issues with non-ResourceLoader requests as well. And because >> we've not been able to reproduce this in production. >> > >> > I don't know enough about Beta cluster or our http caching to know how >> to help. I've already exhausted possibly causes I can think of on my own – >> last month. However I'm happy to join a hangout session at some point to >> consult and help pinpoint the problem from a MediaWiki perspective, we >> should have a prod opsen (knowledgable about Varnish), and a beta devop >> present there as well. >> > >> > -- Timo >> > >> > > On 28 May 2015, at 11:37, Jon Robson <jrob...@wikimedia.org> wrote: >> > > >> > > Firstly apologises for all the browser test spam recently. This is an >> update on what is going on and to give you confidence it is being taken >> seriously. >> > > >> > > I've managed to trace the issue to some kind of caching problem in >> mediawiki core but I'm a bit out of my depth trying to work it out. >> > > >> > > Basically under certain circumstances our code is broken and this is >> why the tests are failing. >> > > >> > > The issue is that we are serving old JavaScript along with new >> JavaScript and these files are naturally incompatible causing the errors as >> they are trying to access old frontend APIs that no longer exist. >> > > >> > > How ResourceLoader deals with caching changed last month and I >> suspect there is a bug in that code. I suspect Timo would be able to help >> me chase down the route cause further. >> > > >> > > In the meantime I'm not sure if there is a way to stop the email spam >> till this gets resolved. One idea I had was that when a test build fails it >> would be great if the email alerts stopped and developers become >> responsible for restarting them. Greg/Antoine/Dan is this possible? There's >> little benefit in continuing to run them every day and emailing about >> faikures until this test is fixed. >> > > >> > > Any help on https://phabricator.wikimedia.org/T100542 much >> appreciated. I'm out of my depth as this issue seems to fall outside the >> good ship readership and more in the ResourceLoader and quality assurance >> sea. >> >> -- >> | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | >> | identi.ca: @greg A18D 1138 8E47 FAC8 1C7D | >> >> _______________________________________________ >> QA mailing list >> QA@lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/qa >> > >
_______________________________________________ QA mailing list QA@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/qa
