[Wikidata-bugs] [Maniphest] [Commented On] T194602: Reoccuring dispatch problems (May 2018)

2018-06-14 Thread gerritbot 
gerritbot added a comment.
Change 439528 merged by Dzahn:
[operations/puppet@production] Switch from 5 mins to 10 mins for wikidata dispatch check

https://gerrit.wikimedia.org/r/439528TASK DETAILhttps://phabricator.wikimedia.org/T194602EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, gerritbotCc: Lucas_Werkmeister_WMDE, gerritbot, Aklapper, hoo, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, lisong, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194602: Reoccuring dispatch problems (May 2018)

2018-06-11 Thread gerritbot 
gerritbot added a comment.
Change 439528 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/puppet@production] Switch from 5 mins to 10 mins for wikidata dispatch check

https://gerrit.wikimedia.org/r/439528TASK DETAILhttps://phabricator.wikimedia.org/T194602EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, gerritbotCc: Lucas_Werkmeister_WMDE, gerritbot, Aklapper, hoo, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, lisong, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194602: Reoccuring dispatch problems (May 2018)

2018-06-07 Thread gerritbot 
gerritbot added a comment.
Change 437785 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Make default dispatchIdleDelay configurable

https://gerrit.wikimedia.org/r/437785TASK DETAILhttps://phabricator.wikimedia.org/T194602EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, gerritbotCc: Lucas_Werkmeister_WMDE, gerritbot, Aklapper, hoo, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, lisong, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194602: Reoccuring dispatch problems (May 2018)

2018-06-06 Thread gerritbot 
gerritbot added a comment.
Change 437785 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] Make default dispatchIdleDelay configurable

https://gerrit.wikimedia.org/r/437785TASK DETAILhttps://phabricator.wikimedia.org/T194602EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, gerritbotCc: Lucas_Werkmeister_WMDE, gerritbot, Aklapper, hoo, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, lisong, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194602: Reoccuring dispatch problems (May 2018)

2018-06-06 Thread Addshore 
Addshore added a comment.
So the monitored lag has now dropped to below the 5 minute alarm, let's see if it stays that way.TASK DETAILhttps://phabricator.wikimedia.org/T194602EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: Lucas_Werkmeister_WMDE, gerritbot, Aklapper, hoo, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, lisong, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194602: Reoccuring dispatch problems (May 2018)

2018-06-06 Thread Addshore 
Addshore added a comment.
A bit more investigation:

Sleep time due to no clients

I added a new graph to grafana based on pre existing data https://grafana.wikimedia.org/dashboard/db/wikidata-dispatch-script

F18831763: image.png

This shows us the full amount of time the script spends doing stuff.

Failure to select clients seems to be the thing slowing the dispatching down, in 1 hour of running 4 dispatches (4 hours of dispatching time) 1 of those 4 hours is spent sleeping because the script failed to find a client.

If I compare this to a 1 hour slot 5 hours ago while we were still running 3 dispatchers

F18831777: image.png

In 1 hours of real time running 3 dispatchers (3 hours of dispatching time) over 1 hour of that time was spent sleeping because the script failed to find a client.

I'm going to revert the switch from 3 to 4 dispatchers and

This partly comes down to the 10 second default sleep for this, perhaps this should be lowered.
Also the dispatch interval, of 25 seconds, could perhaps be lowered, the intention of the interval is to allow for change coalescing during dispatching.

Lag never really decreased to 0

The average lag will always have some high value if we never actually dispatch all changes we have to a client.
Looking at pswiki as an example over the last 3 hours:

06:34:29 Posted 7 changes to pswiki, up to ID 687490812, timestamp 20180606063322. Lag is 67 seconds. Next ID is 687491496.
06:58:38 Posted 3 changes to pswiki, up to ID 687501565, timestamp 20180606065436. Lag is 242 seconds. Next ID is 687503429.
07:22:13 Posted 3 changes to pswiki, up to ID 687512677, timestamp 20180606072047. Lag is 86 seconds. Next ID is 687513358.
07:32:45 Posted 1 changes to pswiki, up to ID 687517674, timestamp 20180606073158. Lag is 47 seconds. Next ID is 687518097.
07:42:08 Posted 3 changes to pswiki, up to ID 687521826, timestamp 20180606074102. Lag is 66 seconds. Next ID is 687522469.
07:51:58 Posted 6 changes to pswiki, up to ID 687527149, timestamp 20180606075117. Lag is 41 seconds. Next ID is 687527523.
08:04:48 Posted 2 changes to pswiki, up to ID 687533358, timestamp 20180606080318. Lag is 90 seconds. Next ID is 687534191.
08:18:53 Posted 3 changes to pswiki, up to ID 687540113, timestamp 20180606081539. Lag is 194 seconds. Next ID is 687541770.
08:34:11 Posted 4 changes to pswiki, up to ID 687548966, timestamp 20180606083240. Lag is 91 seconds. Next ID is 687549634.
08:48:56 Posted 3 changes to pswiki, up to ID 687554842, timestamp 20180606084343. Lag is 313 seconds. Next ID is 687557642.
09:00:55 Posted 3 changes to pswiki, up to ID 687564172, timestamp 20180606085953. Lag is 62 seconds. Next ID is 687564595.
09:15:51 Posted 6 changes to pswiki, up to ID 687572057, timestamp 20180606091341. Lag is 130 seconds. Next ID is 687573257.
09:28:12 Posted 7 changes to pswiki, up to ID 687578481, timestamp 20180606092435. Lag is 217 seconds. Next ID is 687580509.
09:37:04 Posted 6 changes to pswiki, up to ID 687584827, timestamp 20180606093646. Lag is 18 seconds. Next ID is 687584957.
09:45:25 Posted 3 changes to pswiki, up to ID 687588950, timestamp 20180606094344. Lag is 101 seconds. Next ID is 687589974.

When the dispatcher finally got around to dispatching to the client again at each of the below times the dispatch lag was:

06:58:38 - 24 mins + 67 seconds =~ 25 mins
07:22:13 - 24 mins + 242 seconds =~ 28 mins
07:32:45 - 10 mins + 86 seconds =~ 11 mins
07:42:08 - 10 mins + 47 seconds =~ 11 mins
07:51:58 - 9 mins + 66 seconds =~ 10 mins
08:04:48 - 13 mins + 41 seconds =~14 mins

So pswiki for the hour had the average lag of ~15 mins.
The baseline lag can be seen as roughly 1-2 mins.

On sites like enwiki the base lag that never gets dispatched past is more like ~250-600 seconds (even though 420 changes are dispatched during each pass.TASK DETAILhttps://phabricator.wikimedia.org/T194602EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: Lucas_Werkmeister_WMDE, gerritbot, Aklapper, hoo, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, lisong, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194602: Reoccuring dispatch problems (May 2018)

2018-06-06 Thread Addshore 
Addshore added a comment.
After about 20 minutes there is no real change in lag.
I'll leave the patch deployed for the rest of the day and see.TASK DETAILhttps://phabricator.wikimedia.org/T194602EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: Lucas_Werkmeister_WMDE, gerritbot, Aklapper, hoo, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, lisong, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194602: Reoccuring dispatch problems (May 2018)

2018-06-06 Thread Addshore 
Addshore added a comment.
I just deployed the 4 dispatches at a time patch (forgot to tag the ticket).

Let's review over the next hours.TASK DETAILhttps://phabricator.wikimedia.org/T194602EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: Lucas_Werkmeister_WMDE, gerritbot, Aklapper, hoo, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, lisong, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194602: Reoccuring dispatch problems (May 2018)

2018-06-06 Thread gerritbot 
gerritbot added a comment.
Change 435648 merged by jenkins-bot:
[operations/mediawiki-config@master] Wikidata: Always have 4 change dispatchers running

https://gerrit.wikimedia.org/r/435648TASK DETAILhttps://phabricator.wikimedia.org/T194602EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, gerritbotCc: Lucas_Werkmeister_WMDE, gerritbot, Aklapper, hoo, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, lisong, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194602: Reoccuring dispatch problems (May 2018)

2018-06-05 Thread Addshore 
Addshore added a comment.
I'll schedule the dispatching change for some time either this evening or tomorrow.TASK DETAILhttps://phabricator.wikimedia.org/T194602EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: Lucas_Werkmeister_WMDE, gerritbot, Aklapper, hoo, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, lisong, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194602: Reoccuring dispatch problems (May 2018)

2018-06-05 Thread Lucas_Werkmeister_WMDE 
Lucas_Werkmeister_WMDE added a comment.
@hoo what is the current status of this? Can we move forward with having all 4 dispatchers running?TASK DETAILhttps://phabricator.wikimedia.org/T194602EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lucas_Werkmeister_WMDECc: Lucas_Werkmeister_WMDE, gerritbot, Aklapper, hoo, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, lisong, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194602: Reoccuring dispatch problems (May 2018)

2018-05-26 Thread gerritbot 
gerritbot added a comment.
Change 435648 had a related patch set uploaded (by Hoo man; owner: Hoo man):
[operations/mediawiki-config@master] Wikidata: Always have 4 change dispatchers running

https://gerrit.wikimedia.org/r/435648TASK DETAILhttps://phabricator.wikimedia.org/T194602EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: gerritbot, Aklapper, hoo, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, lisong, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194602: Reoccuring dispatch problems (May 2018)

2018-05-13 Thread gerritbot 
gerritbot added a comment.
Change 432778 had a related patch set uploaded (by Hoo man; owner: Hoo man):
[operations/puppet@production] Wikidata: Always have 4 change dispatchers running

https://gerrit.wikimedia.org/r/432778TASK DETAILhttps://phabricator.wikimedia.org/T194602EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: gerritbot, Aklapper, hoo, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs