[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

Ladsgroup added a comment. Thanks! Feel free to reopen in case it started to happen again.TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: daniel, LadsgroupCc: Stashbot, TerraCodes, Jay8g, Liuxinyu970226,

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

Marostegui added a comment. Maybe we should consider this fixed? it has not happened again since Thursday https://grafana.wikimedia.org/dashboard/db/mysql-replication-lag?orgId=1=eqiad%20prometheus%2Fops=now-4d=nowTASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

Marostegui added a comment. In T164173#3533047, @Ladsgroup wrote: The fix is deployed and so I think this task should be closed, can you see anything in the logs that implies it's still happening? It was not easy to see as it would happen all of a sudden. I would prefer to leave it open for a

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

Ladsgroup added a comment. The fix is deployed and so I think this task should be closed, can you see anything in the logs that implies it's still happening?TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To:

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

jcrespo added a comment. @thcipriani @aaron I know something was done yesterday, (thank you!), may I ask for an update of the state, to know if the UBN is still active?TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2017-08-17T16:54:53Z] rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis back to wmf.14 now for T164173TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

jcrespo added a comment. Setting as unbreak now because this is preventing collaborators from editing articles, which causes not only to slowdown volunteers, also causing large frustration among them:

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

Marostegui added a comment. In T164173#3528312, @thcipriani wrote: In T164173#3527782, @Marostegui wrote: @hoo this happened again just now. As we talked during wikimania here is the ping again...do you happen to know if this is about to be released this week in the end? :-) Any fix that

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

thcipriani added a comment. In T164173#3527782, @Marostegui wrote: @hoo this happened again just now. As we talked during wikimania here is the ping again...do you happen to know if this is about to be released this week in the end? :-) Any fix that landed in wikibase before the wmf.14 build

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

Marostegui added a comment. @hoo this happened again just now. As we talked during wikimania here is the ping again...do you happen to know if this is about to be released this week in the end? :-)TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

Marostegui added a comment. @daniel @hoo and myself got together to talk about this yesterday and looks like the patch that is expected to fix this is ready and was already merged. However due to some wikidata releases rollback it is not yet in production, but they expect it to be released in the

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

hoo added a comment. @Marostegui @daniel: Please keep me in the loop when discussing this.TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: daniel, hooCc: Jrbranaa, PokestarFan, Agabi10, gerritbot, Krinkle,

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

Marostegui added a comment. @daniel this happened again yesterday evening (Montreal time, and we got some pages for the slow slaves, not being able to catch up with the master) as you can see on Jaime's comments. I am in Wikimania, so if you like we can try to discuss a bit how to get this

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

jcrespo added a comment. To avoid the continuous lagging on non-directly pooled hosts (passive dc codfw, labs, other hosts replicating on a second tier), I have forced a slowdown of writes to go at the pace of the slowest slaves of eqiad with semisync replication, adding automaticaly a pause of up

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

jcrespo added a comment. I've been told that several thousands of UPDATES Title::invalidateCache per second had caused trouble on s7 over the night, not sure if this is related:

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

gerritbot added a comment. Change 364198 merged by jenkins-bot: [mediawiki/extensions/Wikibase@master] Use multi-page RefreshLinksJobs in WikiPageUpdater https://gerrit.wikimedia.org/r/364198TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

gerritbot added a comment. Change 364205 merged by jenkins-bot: [mediawiki/extensions/Wikibase@master] Use HTMLCacheUpdateJob instead of invalidateCache https://gerrit.wikimedia.org/r/364205TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

aaron added a comment. I commented on the patch, it's a METHOD mismatch problem, so the commit/wait steps don't happen (just the one big one like before).TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To:

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

Krinkle added a comment.Herald added a subscriber: PokestarFan. In T164173#3435975, @gerritbot wrote: Change 364094 merged by jenkins-bot: [mediawiki/extensions/Wikibase@master] Wait for replication after adding RC entries in WikiPageUpdater https://gerrit.wikimedia.org/r/364094 @daniel

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

gerritbot added a comment. Change 364094 merged by jenkins-bot: [mediawiki/extensions/Wikibase@master] Wait for replication after adding RC entries in WikiPageUpdater https://gerrit.wikimedia.org/r/364094TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

aaron added a comment. In T164173#3420723, @daniel wrote: @aaron another question: does RefreshLinksJob also purge the CDN cache automatically? should it? It does update the parser cache... It saves the cache as a convenience in some cases (since the relevant htmlCacheUpdate job uses the

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

daniel added a comment. @aaron another question: does RefreshLinksJob also purge the CDN cache automatically? should it? It does update the parser cache...TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To:

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

gerritbot added a comment. Change 364205 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler): [mediawiki/extensions/Wikibase@master] Use RefreshLinksJob instead of Title:invalidateCache https://gerrit.wikimedia.org/r/364205TASK

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

gerritbot added a comment. Change 364198 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler): [mediawiki/extensions/Wikibase@master] Use multi-page RefreshLinksJobs in WikiPageUpdater https://gerrit.wikimedia.org/r/364198TASK

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

daniel added a comment. @aaron I'm wondering whether WikiPageUpdater::purgeParserCache() is needed at all, if we call WikiPageUpdater::scheduleRefreshLinks() anyway. It's redundant, right? Also, RefreshLinksJob is strange an wonderful... we currently do: $this->jobQueueGroup->push( $job );

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

daniel added a comment. @jcrespo Looking at T169884, it seems that it's an unrelated issue triggered by EmailNotification->notifyOnPageChange(). As far as I understand, notifyOnPageChange should not be triggered by parser cache invalidation, right? If it is, please let me know what code path

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

daniel added a comment. @aaron I'm trying to wrap my head around the code in Title::invalidateCache(), but I could use some guidance. What we want to do is efficiently invalidate the parser cache for a set of pages, not just a single page. I could easily create a AutoCommitUpdate objects that do

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

Ladsgroup added a comment. In T164173#3413805, @Krinkle wrote: PageUpdater::purgeParserCache() -> foreach titles: Title::invalidateCache() My patch adds waitForReplication() to this. PageUpdater::purgeWebCache() -> foreach titles: Title::purgeSquid() I'm not sure about adding

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

gerritbot added a comment. Change 364094 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani): [mediawiki/extensions/Wikibase@master] Wait for replication after macking changes in WikiPageUpdater https://gerrit.wikimedia.org/r/364094TASK

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

jcrespo added a comment. I also wonder why some of those log warnings come from close() and others have the proper commitMasterChanges() bit in the stack trace. Normally, there should be nothing to commit by close() and it is just commits for sanity. We were theorizing the other day on IRC that

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

aaron added a comment. I also wonder why some of those log warnings come from close() and others have the proper commitMasterChanges() bit in the stack trace. Normally, there should be nothing to commit by close() and it is just commits for sanity.TASK

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

Krinkle added a comment. ChangeNotificationJob https://github.com/wikimedia/mediawiki-extensions-Wikibase/blob/6cfd514ee9/client/includes/ChangeNotificationJob.php#L99 ChangeHandler::handleChanges() -> handleChange() -> applyUpdateAction() ChangeHandler::applyUpdateAction(): One of:

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

aaron added a comment. In T164173#3343495, @aaron wrote: @daniel , can you look into the amount of purges happening in ChangeNotification jobs? I don't see any throttling or lag checks on in the job code. /rpc/RunJobs.php?wiki=commonswiki=ChangeNotification=60=300M Expectation (maxAffected

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

jcrespo added a comment. While contention is bad in general- it is the opposite of lag- more contention would create less lag - of course it could be a common source: large updates causing contention on the master, and then lag because the transaction size is large. I wonder if instead of fixing