[Wikidata-bugs] [Maniphest] [Commented On] T173710: Job queue is increasing non-stop

[Krinkle]

Krinkle added a comment.

Added a mitigation section to the task description. Also a summary of the impact of the mitigations so far (based on input from @aaron).

Dashboard: Job Queue Helath
F9232210: Screen Shot 2017-08-31 at 00.00.31.png F9232209: Screen Shot 2017-08-31 at 00.00.18.png
Job queue size down from 10M to a steady ~2M. Before the regression it was a steady between 100K -1M.

Dashboard: Varnish stats
F9232784: Screen Shot 2017-08-31 at 00.15.34.png
Purge rate from production Varnish servers reduced by 2-3X, from 75-100K/s to ~30K/s.

Dashboard: Job Queue Rate for htmlCacheUpdate
F9232855: Screen Shot 2017-08-31 at 00.21.36.png
Queue rate of htmlCacheUpdate back to normal. Deduplication/Superseding optimisation is now working. Execution speed has increased.

TASK DETAIL

https://phabricator.wikimedia.org/T173710

EMAIL PREFERENCES

https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Krinkle
Cc: GWicke, Nemo_bis, Andreasmperu, BBlack, Peachey88, Liuxinyu970226, daniel, Stashbot, Agabi10, Daniel_Mietchen, Harej, XXN, Pasleim, Bugreporter, Sjoerddebruin, Magnus, Mr.Ibrahem, Emijrp, gerritbot, EBernhardson, Esc3300, jcrespo, WMDE-leszek, Jdforrester-WMF, Krinkle, aaron, fgiunchedi, Aklapper, Ladsgroup, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, Vali.matei, Avner, Zppix, debt, Gehel, FloNight, Izno, Eevans, mobrovac, Hardikj, Wikidata-bugs, aude, jayvdb, faidon, Mbch331, Jay8g, jeremyb

_______________________________________________
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs