If Wikimetrics is down because of DB issues, then methinks Quarry will also be down. :(
On Tue, Feb 17, 2015 at 9:40 AM, Amanda Bittaker <[email protected]> wrote: > On that note, Jonathan, do you have SQL queries that return the same > results as the Wikimetrics reports? I tried writing my own for bytes > added, but it's pretty janky and takes forever to return anything in > Quarry. Would you share your wisdom with a poor wayward amateur? > > > On Tue, Feb 17, 2015 at 9:35 AM, Amanda Bittaker <[email protected]> > wrote: > >> Sweet, thanks Jonathan. I added a "Heartbreak" token, because at this >> point I am really far too emotionally attached to Wikimetrics. >> >> On Tue, Feb 17, 2015 at 9:25 AM, Jonathan Morgan <[email protected]> >> wrote: >> >>> Hi Amanda, >>> >>> Here's a ticket you can upvote: https://phabricator.wikimedia.org/T87596 >>> >>> I added a link to this thread to the task. I also added an "Evil Spooky >>> Haunted Tree" token to the task. Because... well it just felt like the >>> right thing to do. >>> >>> - J >>> >>> On Tue, Feb 17, 2015 at 7:58 AM, Dan Andreescu <[email protected] >>> > wrote: >>> >>>> I can't find a specific ticket, Nuria may know of one. In general, >>>> this is the project that LabsDB tickets are tagged with: >>>> https://phabricator.wikimedia.org/tag/wikimedia-labs-infrastructure/ >>>> >>>> On Tue, Feb 17, 2015 at 10:53 AM, Amanda Bittaker < >>>> [email protected]> wrote: >>>> >>>>> Good morning Dan, >>>>> >>>>> Thanks very much for the explanation. Is there a Phabricator task we >>>>> can upvote (award a token?) to make this issue more visible? >>>>> >>>>> As always, we really appreciate your help with this. >>>>> >>>>> Best, >>>>> Amanda >>>>> >>>>> >>>>> >>>>> On Tue, Feb 17, 2015 at 7:20 AM, Dan Andreescu < >>>>> [email protected]> wrote: >>>>> >>>>>> Sorry for the trouble, Amanda. The problem is solely with the >>>>>> underlying database, which we don't maintain. It's a sanitized replica >>>>>> of >>>>>> all the changes being made to all the wikis so it's a fairly complicated >>>>>> piece of infrastructure that sometimes has problems. The folks who >>>>>> maintain it are aware of the issues, but we'll continue representing them >>>>>> until they're solved. >>>>>> >>>>>> On Mon, Feb 16, 2015 at 3:49 PM, Amanda Bittaker < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Oop, thanks for the ping, Nuria. Wikimetrics seems to be working >>>>>>> better now. I still get failures, especially when running three or four >>>>>>> reports in one batch, but the reports work if you rerun them (sometimes >>>>>>> a >>>>>>> couple times.) >>>>>>> >>>>>>> I'm still getting "PENDING"s that turn into "FAILURE"s sometimes, >>>>>>> which I just noticed for the first time last Thursday. Also, sometimes >>>>>>> the >>>>>>> "FAILURE"s change position in the Current Report Inbox list, moving up >>>>>>> or >>>>>>> down a spot. Not sure if that helps diagnose what might be happening... >>>>>>> >>>>>>> In any case, Wikimetrics is mostly functioning but seems to be >>>>>>> having recurring troubles that sometimes blow up to freeze the whole >>>>>>> tool. >>>>>>> It would be great to resolve the troubles before the next explosion--is >>>>>>> there anything I can do to help? Dan H and I still have plenty of >>>>>>> reports >>>>>>> to run, we can keep you updated on the reports ran and failure rate >>>>>>> while >>>>>>> you are fixing, if that would be useful. >>>>>>> >>>>>>> Many thanks, >>>>>>> Amanda >>>>>>> >>>>>>> >>>>>>> On Mon, Feb 16, 2015 at 10:15 AM, Nuria Ruiz <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Ping .... >>>>>>>> >>>>>>>> On Fri, Feb 13, 2015 at 2:19 PM, Nuria Ruiz <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Amanda, >>>>>>>>> >>>>>>>>> Looks like wikimetrics was able to run automatic reports last >>>>>>>>> night w/o big issues, are your reports still failing? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Nuria >>>>>>>>> >>>>>>>>> On Thu, Feb 12, 2015 at 1:42 PM, Amanda Bittaker < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Alright, thanks so much for your help once again, Nuria. >>>>>>>>>> >>>>>>>>>> If there's anything I can do or any information I can contribute, >>>>>>>>>> please don't hesitate to ping me. >>>>>>>>>> >>>>>>>>>> Best, >>>>>>>>>> Amanda >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Feb 12, 2015 at 1:36 PM, Nuria Ruiz <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> DB connections in labs look to be failing, unfortunately I >>>>>>>>>>> think besides asking for help on the labs list there is not much we >>>>>>>>>>> can do >>>>>>>>>>> there. I will start a thread on this regard. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Nuria >>>>>>>>>>> >>>>>>>>>>> On Thu, Feb 12, 2015 at 1:32 PM, Amanda Bittaker < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Thanks so much for the quick response, Nuria. >>>>>>>>>>>> >>>>>>>>>>>> I ran the exact same reports on the same cohort as one of the >>>>>>>>>>>> last batches that were failing. Last time 2/4 of the reports >>>>>>>>>>>> failed, when >>>>>>>>>>>> I reran the individually they succeeded. (But they don't always, >>>>>>>>>>>> I reran >>>>>>>>>>>> one report 3 times this morning before it worked.) This time, my >>>>>>>>>>>> failure >>>>>>>>>>>> rate got worse: 4/4 failed, although they said "PENDING" for a >>>>>>>>>>>> few seconds >>>>>>>>>>>> first, which is new. >>>>>>>>>>>> >>>>>>>>>>>> Is that useful information? Please do let me know what else I >>>>>>>>>>>> can do to help solve this. >>>>>>>>>>>> >>>>>>>>>>>> Thanks again, >>>>>>>>>>>> Amanda >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Feb 12, 2015 at 1:09 PM, Jonathan Morgan < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Thanks Nuria! >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Feb 12, 2015 at 12:57 PM, Nuria Ruiz < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> If so a cohort + report to repro will be most useful. >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Translation:* try to run the exact same reports on the same >>>>>>>>>>>>> cohort again, to see if the same metrics fail. Let us know what >>>>>>>>>>>>> you find. ;) >>>>>>>>>>>>> >>>>>>>>>>>>> Same goes for anyone else who experiences these issues: the >>>>>>>>>>>>> more details we (users) can provide the engineers, the more >>>>>>>>>>>>> effective they >>>>>>>>>>>>> can be at diagnosing and addressing the problems. >>>>>>>>>>>>> >>>>>>>>>>>>> Cheers, >>>>>>>>>>>>> - J >>>>>>>>>>>>> >>>>>>>>>>>>> *for anyone who is not 100% familiar with that hip, new >>>>>>>>>>>>> software engineering lingo >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Nuria >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, Feb 12, 2015 at 12:35 PM, Dan Andreescu < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Recently there was a restart of the labsdb cluster. I'm >>>>>>>>>>>>>>> sorry but I don't have time to check on it, but I bet that's >>>>>>>>>>>>>>> the problem. >>>>>>>>>>>>>>> I'm off tomorrow unfortunately but I'll try to check tomorrow >>>>>>>>>>>>>>> night :( I >>>>>>>>>>>>>>> hope someone else beats me to it. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, Feb 12, 2015 at 3:20 PM, Jonathan Morgan < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> (ping Kevin and Dan A.) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Amanda, I've had some problems with report failures >>>>>>>>>>>>>>>> recently when I ran a few test cohorts. On the same cohort, >>>>>>>>>>>>>>>> when I ran >>>>>>>>>>>>>>>> multiple concurrent reports (say, bytes added, edits, and >>>>>>>>>>>>>>>> pages created), >>>>>>>>>>>>>>>> some would fail and others succeed. It wasn't clear what the >>>>>>>>>>>>>>>> issue was. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> - J >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Thu, Feb 12, 2015 at 12:16 PM, Amanda Bittaker < >>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hello all, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I am getting failures again, both when uploading cohorts >>>>>>>>>>>>>>>>> and running reports. Strangely, it seems the more reports >>>>>>>>>>>>>>>>> you try to run >>>>>>>>>>>>>>>>> in one batch the less likely it is any report will succeed. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Is anyone else having these problems again? Wonderful >>>>>>>>>>>>>>>>> Analytics people, could you please work your magic again? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Many thanks, >>>>>>>>>>>>>>>>> Amanda >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>> Wikimetrics mailing list >>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>> https://lists.wikimedia.org/mailman/listinfo/wikimetrics >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Jonathan T. Morgan >>>>>>>>>>>>>>>> Community Research Lead >>>>>>>>>>>>>>>> Wikimedia Foundation >>>>>>>>>>>>>>>> User:Jmorgan (WMF) >>>>>>>>>>>>>>>> <https://meta.wikimedia.org/wiki/User:Jmorgan_(WMF)> >>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> Wikimetrics mailing list >>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>> https://lists.wikimedia.org/mailman/listinfo/wikimetrics >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Jonathan T. Morgan >>>>>>>>>>>>> Community Research Lead >>>>>>>>>>>>> Wikimedia Foundation >>>>>>>>>>>>> User:Jmorgan (WMF) >>>>>>>>>>>>> <https://meta.wikimedia.org/wiki/User:Jmorgan_(WMF)> >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Wikimetrics mailing list >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>> https://lists.wikimedia.org/mailman/listinfo/wikimetrics >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Wikimetrics mailing list >>>>>>> [email protected] >>>>>>> https://lists.wikimedia.org/mailman/listinfo/wikimetrics >>>>>>> >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Wikimetrics mailing list >>>>>> [email protected] >>>>>> https://lists.wikimedia.org/mailman/listinfo/wikimetrics >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> Wikimetrics mailing list >>>>> [email protected] >>>>> https://lists.wikimedia.org/mailman/listinfo/wikimetrics >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> Wikimetrics mailing list >>>> [email protected] >>>> https://lists.wikimedia.org/mailman/listinfo/wikimetrics >>>> >>>> >>> >>> >>> -- >>> Jonathan T. Morgan >>> Community Research Lead >>> Wikimedia Foundation >>> User:Jmorgan (WMF) <https://meta.wikimedia.org/wiki/User:Jmorgan_(WMF)> >>> [email protected] >>> >>> >>> _______________________________________________ >>> Wikimetrics mailing list >>> [email protected] >>> https://lists.wikimedia.org/mailman/listinfo/wikimetrics >>> >>> >> > > _______________________________________________ > Wikimetrics mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/wikimetrics > > -- Jonathan T. Morgan Community Research Lead Wikimedia Foundation User:Jmorgan (WMF) <https://meta.wikimedia.org/wiki/User:Jmorgan_(WMF)> [email protected]
_______________________________________________ Wikimetrics mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikimetrics
