| EBernhardson added a comment. |
Some background from bblack about the cache purge pipeline:
A) Sometime in the distant past, the way it worked is that when an edit/delete POST request came in, at the end of the request (after sending the response), there's some kind of post-response hook for async processing (before handling another request), and that's where we sent out an immediate, singualr multicast purge towards varnish.
B) Sometime later, at least some of the purging was moved to going through JobQueue. Relatedly, sometime later, someone decided to patch over the rare "lost purge" issue by having MediaWiki send out 3x copies of a given purge, spaced out by several seconds. I think this had to be via the jobqueue (as we wouldn't stall post-response handling to do that) C) I'm not sure, once those things were in play, whether an immediate post-response purge packet was still being sent, in addition to firing off a jobqueue thing for the delayed repeats. or if now all purges only went via jobqueue and never immediate. D) In addition, sometime since 2015 we've started doing a much "better" job of handling templates/transclusions/etc... where the invalidation of a template goes into jobqueue, and then jobqueue recurses through all the dependent objects that got touched and purges those too... we may have gone through multiple steps of "improvement" on this front, generating at each step far larger volumes of purge traffic
And a suggestion from jynus about something that changed around that same time period, not sure how to check if this is related:
13:07 < jynus> interesting, that day is when commons article definition was changed 13:07 < jynus> https://commons.wikimedia.org/wiki/Commons:Village_pump/Archive/2017/07#Should_content_pages_consist_of_galleries_only_or_also_include_File_pages.3F
The increase in purges happens between 8/7 08:00 and 8/8 08:00. Looking at the SAL log there is really nothing interesting in there, train didn't roll forward till later (and then only to group0). Something other than a deployment seems to have triggered this increase.
TASK DETAIL
EMAIL PREFERENCES
To: EBernhardson
Cc: BBlack, Peachey88, Liuxinyu970226, daniel, Stashbot, Agabi10, Daniel_Mietchen, Harej, XXN, Pasleim, Bugreporter, Sjoerddebruin, Magnus, Mr.Ibrahem, Emijrp, gerritbot, EBernhardson, Esc3300, jcrespo, WMDE-leszek, Jdforrester-WMF, Krinkle, aaron, fgiunchedi, Aklapper, Ladsgroup, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, Vali.matei, Avner, Lewizho99, Zppix, Maathavan, debt, Gehel, FloNight, Izno, Wikidata-bugs, aude, jayvdb, faidon, Mbch331, Jay8g, jeremyb
Cc: BBlack, Peachey88, Liuxinyu970226, daniel, Stashbot, Agabi10, Daniel_Mietchen, Harej, XXN, Pasleim, Bugreporter, Sjoerddebruin, Magnus, Mr.Ibrahem, Emijrp, gerritbot, EBernhardson, Esc3300, jcrespo, WMDE-leszek, Jdforrester-WMF, Krinkle, aaron, fgiunchedi, Aklapper, Ladsgroup, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, Vali.matei, Avner, Lewizho99, Zppix, Maathavan, debt, Gehel, FloNight, Izno, Wikidata-bugs, aude, jayvdb, faidon, Mbch331, Jay8g, jeremyb
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
