[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-08-22 Thread Ladsgroup 
Ladsgroup added a comment.
Thanks! Feel free to reopen in case it started to happen again.TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: daniel, LadsgroupCc: Stashbot, TerraCodes, Jay8g, Liuxinyu970226, thcipriani, Jrbranaa, PokestarFan, Agabi10, gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Vali.matei, Minhnv-2809, Zppix, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-08-22 Thread Marostegui 
Marostegui added a comment.
Maybe we should consider this fixed? it has not happened again since Thursday https://grafana.wikimedia.org/dashboard/db/mysql-replication-lag?orgId=1=eqiad%20prometheus%2Fops=now-4d=nowTASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: daniel, MarosteguiCc: Stashbot, TerraCodes, Jay8g, Liuxinyu970226, thcipriani, Jrbranaa, PokestarFan, Agabi10, gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Vali.matei, Minhnv-2809, Zppix, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-08-18 Thread Marostegui 
Marostegui added a comment.

In T164173#3533047, @Ladsgroup wrote:
The fix is deployed and so I think this task should be closed, can you see anything in the logs that implies it's still happening?


It was not easy to see as it would happen all of a sudden.
I would prefer to leave it open for a few days until we can see this is gone for good. It was happening almost every single day lately, so in a few days if it has not happened, I would consider it solved :-)TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: daniel, MarosteguiCc: Stashbot, TerraCodes, Jay8g, Liuxinyu970226, thcipriani, Jrbranaa, PokestarFan, Agabi10, gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Vali.matei, Minhnv-2809, Zppix, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-08-18 Thread Ladsgroup 
Ladsgroup added a comment.
The fix is deployed and so I think this task should be closed, can you see anything in the logs that implies it's still happening?TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: daniel, LadsgroupCc: Stashbot, TerraCodes, Jay8g, Liuxinyu970226, thcipriani, Jrbranaa, PokestarFan, Agabi10, gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Vali.matei, Minhnv-2809, Zppix, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-08-18 Thread jcrespo 
jcrespo added a comment.
@thcipriani @aaron I know something was done yesterday, (thank you!), may I ask for an update of the state, to know if the UBN is still active?TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: daniel, jcrespoCc: Stashbot, TerraCodes, Jay8g, Liuxinyu970226, thcipriani, Jrbranaa, PokestarFan, Agabi10, gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Vali.matei, Minhnv-2809, Zppix, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-08-17 Thread Stashbot 
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2017-08-17T16:54:53Z]  rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis back to wmf.14 now for T164173TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: daniel, StashbotCc: Stashbot, TerraCodes, Jay8g, Liuxinyu970226, thcipriani, Jrbranaa, PokestarFan, Agabi10, gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Vali.matei, Minhnv-2809, Zppix, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-08-17 Thread jcrespo 
jcrespo added a comment.
Setting as unbreak now because this is preventing collaborators from editing articles, which causes not only to slowdown volunteers, also causing large frustration among them: https://grafana.wikimedia.org/dashboard/db/wikidata-edits?refresh=1m=1=1=1502980683784=1502984283784 If 90% of the edits go away because this bug, we should give it top priority. CC #editing-team #community-tech .TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: daniel, jcrespoCc: TerraCodes, Jay8g, Liuxinyu970226, thcipriani, Jrbranaa, PokestarFan, Agabi10, gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Vali.matei, Minhnv-2809, Zppix, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-08-16 Thread Marostegui 
Marostegui added a comment.

In T164173#3528312, @thcipriani wrote:

In T164173#3527782, @Marostegui wrote:
@hoo this happened again just now. As we talked during wikimania here is the ping again...do you happen to know if this is about to be released this week in the end? :-)


Any fix that landed in wikibase before the wmf.14 build (https://github.com/wikimedia/mediawiki-extensions-WikiData/tree/wmf/1.30.0-wmf.14) will roll to group1 wikis with the train happening at 19:00UTC today.


Thanks! Does that mean it would need to appear here?: https://www.mediawiki.org/wiki/MediaWiki_1.30/wmf.14TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: daniel, MarosteguiCc: thcipriani, Jrbranaa, PokestarFan, Agabi10, gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Vali.matei, Minhnv-2809, Zppix, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-08-16 Thread thcipriani 
thcipriani added a comment.

In T164173#3527782, @Marostegui wrote:
@hoo this happened again just now. As we talked during wikimania here is the ping again...do you happen to know if this is about to be released this week in the end? :-)


Any fix that landed in wikibase before the wmf.14 build (https://github.com/wikimedia/mediawiki-extensions-WikiData/tree/wmf/1.30.0-wmf.14) will roll to group1 wikis with the train happening at 19:00UTC today.TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: daniel, thciprianiCc: thcipriani, Jrbranaa, PokestarFan, Agabi10, gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Vali.matei, Minhnv-2809, Zppix, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-08-16 Thread Marostegui 
Marostegui added a comment.
@hoo this happened again just now. As we talked during wikimania here is the ping again...do you happen to know if this is about to be released this week in the end? :-)TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: daniel, MarosteguiCc: Jrbranaa, PokestarFan, Agabi10, gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Vali.matei, Minhnv-2809, Zppix, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-08-13 Thread Marostegui 
Marostegui added a comment.
@daniel @hoo and myself got together to talk about this yesterday and looks like the patch that is expected to fix this is ready and was already merged.

However due to some wikidata releases rollback it is not yet in production, but they expect it to be released in the next few days.

Thank you very much guys!TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: daniel, MarosteguiCc: Jrbranaa, PokestarFan, Agabi10, gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Vali.matei, Minhnv-2809, Zppix, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-08-10 Thread hoo 
hoo added a comment.
@Marostegui @daniel: Please keep me in the loop when discussing this.TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: daniel, hooCc: Jrbranaa, PokestarFan, Agabi10, gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Vali.matei, Minhnv-2809, Zppix, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-08-10 Thread Marostegui 
Marostegui added a comment.
@daniel this happened again yesterday evening (Montreal time, and we got some pages for the slow slaves, not being able to catch up with the master) as you can see on Jaime's comments. 
I am in Wikimania, so if you like we can try to discuss a bit how to get this properly fixed in code, as the hack we have done (throttling this with semisync replication on the slow slaves) might have secondary effects that we are not aware of.

Thanks!TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: daniel, MarosteguiCc: Jrbranaa, PokestarFan, Agabi10, gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Vali.matei, Minhnv-2809, Zppix, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-08-10 Thread jcrespo 
jcrespo added a comment.
To avoid the continuous lagging on non-directly pooled hosts (passive dc codfw, labs, other hosts replicating on a second tier), I have forced a slowdown of writes to go at the pace of the slowest slaves of eqiad with semisync replication, adding automaticaly a pause of up to 100ms between writes. I do not know if that will have an effect such as an increase of the job queue pendding tasks or a slowdown on the rate of edits. CC #performance-team.TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: daniel, jcrespoCc: Jrbranaa, PokestarFan, Agabi10, gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Vali.matei, Minhnv-2809, Zppix, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-08-10 Thread jcrespo 
jcrespo added a comment.
I've been told that several thousands of UPDATES Title::invalidateCache per second had caused trouble on s7 over the night, not sure if this is related:

https://grafana.wikimedia.org/dashboard/db/mysql-replication-lag?orgId=1=eqiad%20prometheus%2Fops=7=1502261251242=1502347651243

UPDATE /* Title::invalidateCache  */  `page` SET page_touched = '20170809234330' WHERE page_id = '645152' AND (page_touched < '20170809234330')
UPDATE /* Title::invalidateCache  */  `page` SET page_touched = '20170809234330' WHERE page_id = '645704' AND (page_touched < '20170809234330')
UPDATE /* Title::invalidateCache  */  `page` SET page_touched = '20170809234330' WHERE page_id = '646071' AND (page_touched < '20170809234330')
UPDATE /* Title::invalidateCache  */  `page` SET page_touched = '20170809234330' WHERE page_id = '646352' AND (page_touched < '20170809234330')
UPDATE /* Title::invalidateCache  */  `page` SET page_touched = '20170809234330' WHERE page_id = '646766' AND (page_touched < '20170809234330')
UPDATE /* Title::invalidateCache  */  `page` SET page_touched = '20170809234330' WHERE page_id = '647384' AND (page_touched < '20170809234330')
UPDATE /* Title::invalidateCache  */  `page` SET page_touched = '20170809234330' WHERE page_id = '649512' AND (page_touched < '20170809234330')
UPDATE /* Title::invalidateCache  */  `page` SET page_touched = '20170809234330' WHERE page_id = '649759' AND (page_touched < '20170809234330')
UPDATE /* Title::invalidateCache  */  `page` SET page_touched = '20170809234330' WHERE page_id = '649849' AND (page_touched < '20170809234330')TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: daniel, jcrespoCc: Jrbranaa, PokestarFan, Agabi10, gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Vali.matei, Minhnv-2809, Zppix, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-08-01 Thread gerritbot 
gerritbot added a comment.
Change 364198 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Use multi-page RefreshLinksJobs in WikiPageUpdater

https://gerrit.wikimedia.org/r/364198TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: daniel, gerritbotCc: PokestarFan, Agabi10, gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, Vali.matei, Lewizho99, Minhnv-2809, Zppix, Maathavan, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-08-01 Thread gerritbot 
gerritbot added a comment.
Change 364205 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Use HTMLCacheUpdateJob instead of invalidateCache

https://gerrit.wikimedia.org/r/364205TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: daniel, gerritbotCc: PokestarFan, Agabi10, gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, Vali.matei, Lewizho99, Minhnv-2809, Zppix, Maathavan, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-07-22 Thread aaron 
aaron added a comment.
I commented on the patch, it's a METHOD mismatch problem, so the commit/wait steps don't happen (just the one big one like before).TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: daniel, aaronCc: PokestarFan, Agabi10, gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, Vali.matei, Lewizho99, Minhnv-2809, Zppix, Maathavan, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-07-21 Thread Krinkle 
Krinkle added a comment.Herald added a subscriber: PokestarFan.

In T164173#3435975, @gerritbot wrote:
Change 364094 merged by jenkins-bot:
 [mediawiki/extensions/Wikibase@master] Wait for replication after adding RC entries in WikiPageUpdater

https://gerrit.wikimedia.org/r/364094


@daniel @Ladsgroup This caused an error spike in Logstash:
https://logstash.wikimedia.org/goto/78333176c796825764e0d0c34ab43815
F8808868: Screen Shot 2017-07-21 at 19.27.46.png

Wikimedia\Rdbms\LBFactory::getEmptyTransactionTicket: Wikibase\Client\Changes\WikiPageUpdater::injectRCRecords does not have outer scope.
#0 /srv/mediawiki/php-1.30.0-wmf.10/extensions/Wikidata/extensions/Wikibase/client/includes/Changes/WikiPageUpdater.php(161): Wikimedia\Rdbms\LBFactory->getEmptyTransactionTicket()
#1 /srv/mediawiki/php-1.30.0-wmf.10/extensions/Wikidata/extensions/Wikibase/client/includes/Changes/ChangeHandler.php(250): Wikibase\Client\Changes\WikiPageUpdater->injectRCRecords()
#2 /srv/mediawiki/php-1.30.0-wmf.10/extensions/Wikidata/extensions/Wikibase/client/includes/Changes/ChangeHandler.php(169): Wikibase\Client\Changes\ChangeHandler->applyUpdateAction()
#3 /srv/mediawiki/php-1.30.0-wmf.10/extensions/Wikidata/extensions/Wikibase/client/includes/Changes/ChangeHandler.php(137): Wikibase\Client\Changes\ChangeHandler->handleChange()
#4 /srv/mediawiki/php-1.30.0-wmf.10/extensions/Wikidata/extensions/Wikibase/client/includes/ChangeNotificationJob.php(99): Wikibase\Client\Changes\ChangeHandler->handleChanges()
#5 /srv/mediawiki/php-1.30.0-wmf.10/includes/jobqueue/JobRunner.php(293): Wikibase\Client\ChangeNotificationJob->run()
#6 /srv/mediawiki/php-1.30.0-wmf.10/includes/jobqueue/JobRunner.php(193): JobRunner->executeJob()
#7 /srv/mediawiki/rpc/RunJobs.php(47): JobRunner->run()
#8 {main}TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: daniel, KrinkleCc: PokestarFan, Agabi10, gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, Vali.matei, Lewizho99, Minhnv-2809, Zppix, Maathavan, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-07-13 Thread gerritbot 
gerritbot added a comment.
Change 364094 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Wait for replication after adding RC entries in WikiPageUpdater

https://gerrit.wikimedia.org/r/364094TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: daniel, gerritbotCc: Agabi10, gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, Vali.matei, Lewizho99, Minhnv-2809, Zppix, Maathavan, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-07-11 Thread aaron 
aaron added a comment.

In T164173#3420723, @daniel wrote:
@aaron another question: does RefreshLinksJob also purge the CDN cache automatically? should it? It does update the parser cache...


It saves the cache as a convenience in some cases (since the relevant htmlCacheUpdate job uses the rootJobTimestamp in setting page_touched, so some cache misses can be avoided by doing so). The only CDN purges refreshLinks does is minor thinks like file description pages (since they list usage links).TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: daniel, aaronCc: Agabi10, gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, Vali.matei, Lewizho99, Minhnv-2809, Zppix, Maathavan, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-07-10 Thread daniel 
daniel added a comment.
@aaron another question: does RefreshLinksJob also purge the CDN cache automatically? should it? It does update the parser cache...TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: danielCc: Agabi10, gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, Vali.matei, Lewizho99, Minhnv-2809, Zppix, Maathavan, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-07-10 Thread gerritbot 
gerritbot added a comment.
Change 364205 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/extensions/Wikibase@master] Use RefreshLinksJob instead of Title:invalidateCache

https://gerrit.wikimedia.org/r/364205TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: daniel, gerritbotCc: Agabi10, gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, Vali.matei, Lewizho99, Minhnv-2809, Zppix, Maathavan, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-07-10 Thread gerritbot 
gerritbot added a comment.
Change 364198 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/extensions/Wikibase@master] Use multi-page RefreshLinksJobs in WikiPageUpdater

https://gerrit.wikimedia.org/r/364198TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ladsgroup, gerritbotCc: gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, Vali.matei, Lewizho99, Minhnv-2809, Zppix, Maathavan, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-07-10 Thread daniel 
daniel added a comment.
@aaron I'm wondering whether WikiPageUpdater::purgeParserCache() is needed at all, if we call WikiPageUpdater::scheduleRefreshLinks() anyway. It's redundant, right?

Also, RefreshLinksJob is strange an wonderful... we currently do:

			$this->jobQueueGroup->push( $job );
			$this->jobQueueGroup->deduplicateRootJob( $job );

...with a separate job for each title. But I'm guessing we should use a single job for all titles, and then just do

			$this->jobQueueGroup->lazyPush( $job );

Is that right?TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ladsgroup, danielCc: gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, Vali.matei, Lewizho99, Minhnv-2809, Zppix, Maathavan, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-07-10 Thread daniel 
daniel added a comment.
@jcrespo Looking at T169884, it seems that it's an unrelated issue triggered by  EmailNotification->notifyOnPageChange(). As far as I understand, notifyOnPageChange should not be triggered by parser cache invalidation, right? If it is, please let me know what code path leads from UpdateNotificationJob or WikiPageUpdater to notifyOnPageChange().TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ladsgroup, danielCc: gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, Vali.matei, Lewizho99, Minhnv-2809, Zppix, Maathavan, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-07-10 Thread daniel 
daniel added a comment.
@aaron I'm trying to wrap my head around the code in Title::invalidateCache(), but I could use some guidance. What we want to do is efficiently invalidate the parser cache for a set of pages, not just a single page. I could easily create a AutoCommitUpdate objects that do this in batches of, say, 100 titles. But it's unclear to me how this interacts with automatic transactions. We'd want to wait for replication after each batch, but we can only do that if the transaction was committed. Should I start a new transaction in each AutoCommitUpdate object? Or should I use commitAndWaitForReplication() in the callback?TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ladsgroup, danielCc: gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, Vali.matei, Lewizho99, Minhnv-2809, Zppix, Maathavan, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-07-08 Thread Ladsgroup 
Ladsgroup added a comment.

In T164173#3413805, @Krinkle wrote:

PageUpdater::purgeParserCache() -> foreach titles: Title::invalidateCache()



My patch adds waitForReplication() to this.


PageUpdater::purgeWebCache() -> foreach titles: Title::purgeSquid()


I'm not sure about adding waitForReplication here, I checked the code and couldn't find any place that it changes database. Correct me if I'm wrong


PageUpdater::scheduleRefreshLinks() -> foreach titles: JobQueueGroup->push( new RefreshLinksJob )


RefreshLinksJob has waitForReplication() already


PageUpdater::injectRCRecords() -> foreach titles: Recentchange::save()


My patch adds waitForReplication() to this.TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: LadsgroupCc: gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, Vali.matei, Lewizho99, Minhnv-2809, Zppix, Maathavan, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-07-08 Thread gerritbot 
gerritbot added a comment.
Change 364094 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/extensions/Wikibase@master] Wait for replication after macking changes in WikiPageUpdater

https://gerrit.wikimedia.org/r/364094TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ladsgroup, gerritbotCc: gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Vali.matei, Minhnv-2809, Zppix, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-07-07 Thread jcrespo 
jcrespo added a comment.
I also wonder why some of those log warnings come from close() and others have the proper commitMasterChanges() bit in the stack trace. Normally, there should be nothing to commit by close() and it is just commits for sanity.

We were theorizing the other day on IRC that the connection was trying to close a connection that was already closed. Maybe the query killer kills long running queries and the connection handler doesn't like that?TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Vali.matei, Minhnv-2809, Zppix, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-07-06 Thread aaron 
aaron added a comment.
I also wonder why some of those log warnings come from close() and others have the proper commitMasterChanges() bit in the stack trace. Normally, there should be nothing to commit by close() and it is just commits for sanity.TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Vali.matei, Minhnv-2809, Zppix, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-07-06 Thread Krinkle 
Krinkle added a comment.
ChangeNotificationJob https://github.com/wikimedia/mediawiki-extensions-Wikibase/blob/6cfd514ee9/client/includes/ChangeNotificationJob.php#L99


ChangeHandler::handleChanges() -> handleChange() -> applyUpdateAction()
ChangeHandler::applyUpdateAction(): One of: purgeParserCache(), purgeWebCache(), scheduleRefreshLinks(), or injectRCRecords().
Defined at https://github.com/wikimedia/mediawiki-extensions-Wikibase/blob/6cfd514ee99/client/includes/Changes/WikiPageUpdater.php#L74-L147
PageUpdater::purgeParserCache() -> foreach titles: Title::invalidateCache()
PageUpdater::purgeWebCache() -> foreach titles: Title::purgeSquid()
PageUpdater::scheduleRefreshLinks() -> foreach titles: JobQueueGroup->push( new RefreshLinksJob )
PageUpdater::injectRCRecords() -> foreach titles: Recentchange::save()




Both the recentchanges INSERT and the page.page_touched UPDATE queries come from this job.TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: KrinkleCc: Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Vali.matei, Minhnv-2809, Zppix, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-07-06 Thread aaron 
aaron added a comment.

In T164173#3343495, @aaron wrote:
@daniel , can you look into the amount of purges happening in ChangeNotification jobs? I don't see any throttling or lag checks on in the job code.




/rpc/RunJobs.php?wiki=commonswiki=ChangeNotification=60=300M

Expectation (maxAffected <= 500) by JobRunner::run not met (actual: 89936):
[transaction f7dd6f writes to 10.64.48.23 (commonswiki)]
#0 /srv/mediawiki/php-1.30.0-wmf.7/includes/libs/rdbms/TransactionProfiler.php(294): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated()
#1 /srv/mediawiki/php-1.30.0-wmf.7/includes/libs/rdbms/database/Database.php(2888): Wikimedia\Rdbms\TransactionProfiler->transactionWritingOut()
#2 /srv/mediawiki/php-1.30.0-wmf.7/includes/libs/rdbms/database/Database.php(756): Wikimedia\Rdbms\Database->commit()
#3 /srv/mediawiki/php-1.30.0-wmf.7/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1084): Wikimedia\Rdbms\Database->close()
#4 (): Closure$Wikimedia\Rdbms\LoadBalancer::closeAll()
#5 /srv/mediawiki/php-1.30.0-wmf.7/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1485): call_user_func_array()
#6 /srv/mediawiki/php-1.30.0-wmf.7/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1085): Wikimedia\Rdbms\LoadBalancer->forEachOpenConnection()
#7 /srv/mediawiki/php-1.30.0-wmf.7/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1076): Wikimedia\Rdbms\LoadBalancer->closeAll()
#8 /srv/mediawiki/php-1.30.0-wmf.7/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1665): Wikimedia\Rdbms\LoadBalancer->disable()
#9 (): Wikimedia\Rdbms\LoadBalancer->__destruct()
#10 {main}

If those are page_touched updates, that would a pretty large number of purges to make in one job at once.TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Vali.matei, Minhnv-2809, Zppix, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis

2017-05-26 Thread jcrespo 
jcrespo added a comment.
While contention is bad in general- it is the opposite of lag- more contention would create less lag - of course it could be a common  source: large updates causing contention on the master, and then lag because the transaction size is large.

I wonder if instead of fixing each individual query, we should wrap database writes with some kind of monitoring layer that would measure (with heavy caching, of course) the db state and throttle things automatically. I could even increase the pt-heartbeat rate to have more accurate lag measuring to go into sub-second metrics. All of this is probably off-topic here.TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaron, jcrespoCc: Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Vali.matei, Minhnv-2809, Zppix, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs