| Addshore added a comment. |
Thanks for filing this @hoo, I was going to write something up about this today!
While trying to get rid of some of the large backlog of changes to dispatch on Thursday I ran a copy of the dispatch script on terbium, however realised that I has incorrect params (I wanted to change them) so Ctrl + C'd out of the script.
As far as I was able to tell this left any open locks from that run of the script in place.
You can see this @ https://grafana.wikimedia.org/dashboard/db/wikidata-dispatch?orgId=1&from=1508398594143&to=1508458610316.
@ roughly 21:35 I kill some runs of the script and they leave locks open, even the freshest lag starts to rise at this point.
@ roughly 22:35 the locks TTL expires and the scripts start running again.
It also took me roughly the whole hour to figure out that locker were the issue and figure out how to manually remove the locks through eval.php hence why I couldn't fix my screw-up sooner.
Dispatch changes currently runs with a max time of 540 seconds ( 9 mins ).
We should check if it is possible to set the log TTL in the script run, then we can simply use the --max-time parameter.
Cc: Ladsgroup, daniel, Lydia_Pintscher, Addshore, Aklapper, hoo, GoranSMilovanovic, Jayprakash12345, QZanden, Zoranzoki21, DatGuy, Devwaker, Urbanecm, JEumerus, Tulsi_Bhagat, Wong128hk, Luke081515, Simeondahl, biplabanand, Wikidata-bugs, Snowolf, aude, Dcljr, Jdforrester-WMF, Matanya, Mbch331, Rxy, Jay8g, Krenair
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
