[Wikidata-bugs] [Maniphest] [Commented On] T178652: Wikidata dispatchers should use a LockManager with a short TTL

2018-09-25 Thread Addshore 
Addshore added a comment.
It looks like today we can see how useful this is.
A lock for a wiki somehow got stuck, but rather than taking 2 hours to recover, the recovery only took 15 mins.

F26195467: image.pngTASK DETAILhttps://phabricator.wikimedia.org/T178652EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: Niharika, Stashbot, aaron, gerritbot, Ladsgroup, daniel, Lydia_Pintscher, Addshore, Aklapper, hoo, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Jayprakash12345, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Zoranzoki21, LawExplorer, Lewizho99, Maathavan, DatGuy, Devwaker, Niklitov, Urbanecm, JEumerus, Jonas, Tulsi_Bhagat, Wong128hk, Luke081515, SimmeD, Wikidata-bugs, Snowolf, aude, Dcljr, Jdforrester-WMF, Matanya, Mbch331, Rxy, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T178652: Wikidata dispatchers should use a LockManager with a short TTL

2018-07-26 Thread gerritbot 
gerritbot added a comment.
Change 395969 merged by jenkins-bot:
[operations/mediawiki-config@master] Use new wikibase dispatch lock manager on wikidatawiki

https://gerrit.wikimedia.org/r/395969TASK DETAILhttps://phabricator.wikimedia.org/T178652EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, gerritbotCc: Niharika, Stashbot, aaron, gerritbot, Ladsgroup, daniel, Lydia_Pintscher, Addshore, Aklapper, hoo, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Jayprakash12345, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Zoranzoki21, LawExplorer, Lewizho99, Maathavan, DatGuy, Devwaker, Niklitov, Urbanecm, JEumerus, Jonas, Tulsi_Bhagat, Wong128hk, Luke081515, SimmeD, Wikidata-bugs, Snowolf, aude, Dcljr, Jdforrester-WMF, Matanya, Mbch331, Rxy, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T178652: Wikidata dispatchers should use a LockManager with a short TTL

2018-07-12 Thread Stashbot 
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-07-12T18:32:05Z]  Synchronized wmf-config/: Wikidata dispatch, Use a LockManager with short TTL for testwikidata T178652 (duration: 00m 51s)TASK DETAILhttps://phabricator.wikimedia.org/T178652EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: StashbotCc: Stashbot, aaron, gerritbot, Ladsgroup, daniel, Lydia_Pintscher, Addshore, Aklapper, hoo, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Jayprakash12345, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Zoranzoki21, LawExplorer, Lewizho99, Maathavan, DatGuy, Devwaker, Niklitov, Urbanecm, JEumerus, Jonas, Tulsi_Bhagat, Wong128hk, Luke081515, SimmeD, Wikidata-bugs, Snowolf, aude, Dcljr, Jdforrester-WMF, Matanya, Mbch331, Rxy, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T178652: Wikidata dispatchers should use a LockManager with a short TTL

2018-07-12 Thread gerritbot 
gerritbot added a comment.
Change 395967 merged by jenkins-bot:
[operations/mediawiki-config@master] Wikidata dispatch, Use a LockManager with short TTL for testwikidata

https://gerrit.wikimedia.org/r/395967TASK DETAILhttps://phabricator.wikimedia.org/T178652EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: aaron, gerritbot, Ladsgroup, daniel, Lydia_Pintscher, Addshore, Aklapper, hoo, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Jayprakash12345, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Zoranzoki21, LawExplorer, Lewizho99, Maathavan, DatGuy, Devwaker, Niklitov, Urbanecm, JEumerus, Jonas, Tulsi_Bhagat, Wong128hk, Luke081515, SimmeD, Wikidata-bugs, Snowolf, aude, Dcljr, Jdforrester-WMF, Matanya, Mbch331, Rxy, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T178652: Wikidata dispatchers should use a LockManager with a short TTL

2018-06-20 Thread hoo 
hoo added a comment.

In T178652#3820967, @Addshore wrote:
I'm going to first make a change in the dispatch script to check a mediawiki config var to see if it should actually run or not and backport this first.


This has been done (and deployed) by now, so I guess we can move on here whenever we want.TASK DETAILhttps://phabricator.wikimedia.org/T178652EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, hooCc: aaron, gerritbot, Ladsgroup, daniel, Lydia_Pintscher, Addshore, Aklapper, hoo, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Jayprakash12345, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Zoranzoki21, LawExplorer, Lewizho99, Maathavan, DatGuy, Devwaker, Niklitov, Urbanecm, JEumerus, Tulsi_Bhagat, Wong128hk, Luke081515, SimmeD, Wikidata-bugs, Snowolf, aude, Dcljr, Jdforrester-WMF, Matanya, Mbch331, Rxy, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T178652: Wikidata dispatchers should use a LockManager with a short TTL

2017-12-07 Thread Addshore 
Addshore added a comment.
I'm going to first make a change in the dispatch script to check a mediawiki config var to see if it should actually run or not and backport this first.

This will mean that I can:


stop dispatching
Change the lock manager over
enable dispatching


Without trying to coordinate this with changes in operations-puppet.

This is needed otherwise locks will be missed. Old runing scripts will be using the old lock manager and the new ones will get the new lock manager with no locks.TASK DETAILhttps://phabricator.wikimedia.org/T178652EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: gerritbot, Ladsgroup, daniel, Lydia_Pintscher, Addshore, Aklapper, hoo, Lahi, Gq86, Baloch007, Lordiis, GoranSMilovanovic, Adik2382, Jayprakash12345, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Zoranzoki21, Lewizho99, Maathavan, DatGuy, Devwaker, Urbanecm, JEumerus, Tulsi_Bhagat, Wong128hk, Luke081515, SimmeD, biplabanand, Wikidata-bugs, Snowolf, aude, Dcljr, Jdforrester-WMF, Matanya, Mbch331, Rxy, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T178652: Wikidata dispatchers should use a LockManager with a short TTL

2017-12-07 Thread gerritbot 
gerritbot added a comment.
Change 395969 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] Use new wikibase dispatch lock manager on wikidatawiki

https://gerrit.wikimedia.org/r/395969TASK DETAILhttps://phabricator.wikimedia.org/T178652EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: gerritbot, Ladsgroup, daniel, Lydia_Pintscher, Addshore, Aklapper, hoo, Lahi, Gq86, Baloch007, Lordiis, GoranSMilovanovic, Adik2382, Jayprakash12345, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Zoranzoki21, Lewizho99, Maathavan, DatGuy, Devwaker, Urbanecm, JEumerus, Tulsi_Bhagat, Wong128hk, Luke081515, SimmeD, biplabanand, Wikidata-bugs, Snowolf, aude, Dcljr, Jdforrester-WMF, Matanya, Mbch331, Rxy, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T178652: Wikidata dispatchers should use a LockManager with a short TTL

2017-12-07 Thread gerritbot 
gerritbot added a comment.
Change 395967 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] Create a LockManager for WikidataDispatch with short TTL

https://gerrit.wikimedia.org/r/395967TASK DETAILhttps://phabricator.wikimedia.org/T178652EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: gerritbot, Ladsgroup, daniel, Lydia_Pintscher, Addshore, Aklapper, hoo, Lahi, Gq86, GoranSMilovanovic, Jayprakash12345, QZanden, Zoranzoki21, DatGuy, Devwaker, Urbanecm, JEumerus, Tulsi_Bhagat, Wong128hk, Luke081515, SimmeD, biplabanand, Wikidata-bugs, Snowolf, aude, Dcljr, Jdforrester-WMF, Matanya, Mbch331, Rxy, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T178652: Wikidata dispatchers should use a LockManager with a short TTL

2017-10-30 Thread daniel 
daniel added a comment.
@hoo Sounds like we should just go ahead and do this! Don't forget to change dispatchingLockManager accordingly.TASK DETAILhttps://phabricator.wikimedia.org/T178652EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: danielCc: Ladsgroup, daniel, Lydia_Pintscher, Addshore, Aklapper, hoo, Lahi, GoranSMilovanovic, Jayprakash12345, QZanden, Zoranzoki21, DatGuy, Devwaker, Urbanecm, JEumerus, Tulsi_Bhagat, Wong128hk, Luke081515, SimmeD, biplabanand, Wikidata-bugs, Snowolf, aude, Dcljr, Jdforrester-WMF, Matanya, Mbch331, Rxy, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T178652: Wikidata dispatchers should use a LockManager with a short TTL

2017-10-24 Thread Addshore 
Addshore added a comment.
Thanks for filing this @hoo, I was going to write something up about this today!

While trying to get rid of some of the large backlog of changes to dispatch on Thursday I ran a copy of the dispatch script on terbium, however realised that I has incorrect params (I wanted to change them) so Ctrl + C'd out of the script.
As far as I was able to tell this left any open locks from that run of the script in place.

You can see this @ https://grafana.wikimedia.org/dashboard/db/wikidata-dispatch?orgId=1=1508398594143=1508458610316.
@ roughly 21:35 I kill some runs of the script and they leave locks open, even the freshest lag starts to rise at this point.
@ roughly 22:35 the locks TTL expires and the scripts start running again.
It also took me roughly the whole hour to figure out that locker were the issue and figure out how to manually remove the locks through eval.php hence why I couldn't fix my screw-up sooner.

Dispatch changes currently runs with a max time of 540 seconds ( 9 mins ).
We should check if it is possible to set the log TTL in the script run, then we can simply use the --max-time parameter.TASK DETAILhttps://phabricator.wikimedia.org/T178652EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: Ladsgroup, daniel, Lydia_Pintscher, Addshore, Aklapper, hoo, GoranSMilovanovic, Jayprakash12345, QZanden, Zoranzoki21, DatGuy, Devwaker, Urbanecm, JEumerus, Tulsi_Bhagat, Wong128hk, Luke081515, Simeondahl, biplabanand, Wikidata-bugs, Snowolf, aude, Dcljr, Jdforrester-WMF, Matanya, Mbch331, Rxy, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs