https://bugzilla.wikimedia.org/show_bug.cgi?id=60348

       Web browser: ---
            Bug ID: 60348
           Summary: Handle large bursts of jobs more elegantly
           Product: MediaWiki
           Version: 1.23-git
          Hardware: All
                OS: All
            Status: NEW
          Severity: normal
          Priority: Unprioritized
         Component: JobQueue
          Assignee: wikibugs-l@lists.wikimedia.org
          Reporter: innocentkil...@gmail.com
                CC: aschulz4...@gmail.com, tstarl...@wikimedia.org
    Classification: Unclassified
   Mobile Platform: ---

The job queue is way more awesome than it was 2 years ago. With the improved
code + redis architecture, it's incredibly reliable and we're doing way more
jobs than ever before.

We tend to keep up with the small day-to-day jobs perfectly. Most queues are
near empty on enwiki most of the time, or in the case of
cirrus/htmlCache/linksUpdate jobs, maybe a few hundred/thousand at a time. No
big deal.

What we do *not* handle well is a large burst of jobs. Someone edits a super
high use template, we reindex all of enwiki or commons in Cirrus, anything. We
end up with millions of jobs and it takes weeks to clear the backlog without
manual intervention.

It would be nice to do something better in this case. I have no clue what this
better thing may be.

-- 
You are receiving this mail because:
You are the assignee for the bug.
You are on the CC list for the bug.
_______________________________________________
Wikibugs-l mailing list
Wikibugs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikibugs-l

Reply via email to