https://bugzilla.wikimedia.org/show_bug.cgi?id=67543
Bug ID: 67543
Summary: Performance of Recurrent Reports
Product: Analytics
Version: unspecified
Hardware: All
OS: All
Status: NEW
Severity: normal
Priority: Unprioritized
Component: Wikimetrics
Assignee: [email protected]
Reporter: [email protected]
CC: [email protected],
[email protected], [email protected],
[email protected], [email protected],
[email protected], [email protected]
Web browser: ---
Mobile Platform: ---
While testing how many instances of recurring reports we could spawn as part of
the daily run, we discovered a lack of understanding around wikimetrics
database performance and queue concurrency. We want to address that as
follows:
* find out why, if we run more than about 100 report instances at the same
time, we get mysql errors about QueuePool limit.
* if this QueuePool error is completely fixable with configuration, great
* if configuration doesn't completely solve our problem, look into throttling
the number of parallel tasks that we can submit to the queue from the
recurrent_reports() function. Right now, it's unlimited and that's not great.
Celery has lots of constructs like chords and groups that can help us batch
things up better.
--
You are receiving this mail because:
You are the assignee for the bug.
You are on the CC list for the bug.
_______________________________________________
Wikibugs-l mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
