| EBernhardson added a subscriber: BBlack. EBernhardson added a comment. |
I've been doing some testing with the HTMLCacheUpdate jobs against relatively low traffic wikis with high numbers of jobs (viwki and srwiki with 2M and 600k page views/day respectively). It seems like almost the entirety of the time being taken is to allow for throttling. The throttling for this is currently set to 30 'work items' per second. I'm not entirely sure how that works out, but it looks like that's 30 work items per server running jobs(of which i count 15 in eqiad)? Giving us roughly 450 items per second, 27k per minute, 1.6M per hour, and 38M per day.
Talking to @BBlack there is also a coninciding increase in purge jobs: https://grafana.wikimedia.org/dashboard/db/varnish-aggregate-client-status-codes?orgId=1&var-site=All&var-cache_type=upload&var-status_type=1&var-status_type=2&var-status_type=3&var-status_type=4&var-status_type=5&from=now-90d&to=now
Basically around 8/8, the same time the job queue started growing, purge rates increased dramatically and stayed there. Somehow we are creating significantly more purges and they are probably linked to something deloyed around 8/8.
Cc: BBlack, Peachey88, Liuxinyu970226, daniel, Stashbot, Agabi10, Daniel_Mietchen, Harej, XXN, Pasleim, Bugreporter, Sjoerddebruin, Magnus, Mr.Ibrahem, Emijrp, gerritbot, EBernhardson, Esc3300, jcrespo, WMDE-leszek, Jdforrester-WMF, Krinkle, aaron, fgiunchedi, Aklapper, Ladsgroup, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, Vali.matei, Avner, Lewizho99, Zppix, Maathavan, debt, Gehel, FloNight, Izno, Wikidata-bugs, aude, jayvdb, faidon, Mbch331, Jay8g, jeremyb
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
