| Addshore added a comment. |
A bit more investigation:
Sleep time due to no clients
I added a new graph to grafana based on pre existing data https://grafana.wikimedia.org/dashboard/db/wikidata-dispatch-script
This shows us the full amount of time the script spends doing stuff.
Failure to select clients seems to be the thing slowing the dispatching down, in 1 hour of running 4 dispatches (4 hours of dispatching time) 1 of those 4 hours is spent sleeping because the script failed to find a client.
If I compare this to a 1 hour slot 5 hours ago while we were still running 3 dispatchers
In 1 hours of real time running 3 dispatchers (3 hours of dispatching time) over 1 hour of that time was spent sleeping because the script failed to find a client.
I'm going to revert the switch from 3 to 4 dispatchers and
This partly comes down to the 10 second default sleep for this, perhaps this should be lowered.
Also the dispatch interval, of 25 seconds, could perhaps be lowered, the intention of the interval is to allow for change coalescing during dispatching.
Lag never really decreased to 0
The average lag will always have some high value if we never actually dispatch all changes we have to a client.
Looking at pswiki as an example over the last 3 hours:
06:34:29 Posted 7 changes to pswiki, up to ID 687490812, timestamp 20180606063322. Lag is 67 seconds. Next ID is 687491496. 06:58:38 Posted 3 changes to pswiki, up to ID 687501565, timestamp 20180606065436. Lag is 242 seconds. Next ID is 687503429. 07:22:13 Posted 3 changes to pswiki, up to ID 687512677, timestamp 20180606072047. Lag is 86 seconds. Next ID is 687513358. 07:32:45 Posted 1 changes to pswiki, up to ID 687517674, timestamp 20180606073158. Lag is 47 seconds. Next ID is 687518097. 07:42:08 Posted 3 changes to pswiki, up to ID 687521826, timestamp 20180606074102. Lag is 66 seconds. Next ID is 687522469. 07:51:58 Posted 6 changes to pswiki, up to ID 687527149, timestamp 20180606075117. Lag is 41 seconds. Next ID is 687527523. 08:04:48 Posted 2 changes to pswiki, up to ID 687533358, timestamp 20180606080318. Lag is 90 seconds. Next ID is 687534191. 08:18:53 Posted 3 changes to pswiki, up to ID 687540113, timestamp 20180606081539. Lag is 194 seconds. Next ID is 687541770. 08:34:11 Posted 4 changes to pswiki, up to ID 687548966, timestamp 20180606083240. Lag is 91 seconds. Next ID is 687549634. 08:48:56 Posted 3 changes to pswiki, up to ID 687554842, timestamp 20180606084343. Lag is 313 seconds. Next ID is 687557642. 09:00:55 Posted 3 changes to pswiki, up to ID 687564172, timestamp 20180606085953. Lag is 62 seconds. Next ID is 687564595. 09:15:51 Posted 6 changes to pswiki, up to ID 687572057, timestamp 20180606091341. Lag is 130 seconds. Next ID is 687573257. 09:28:12 Posted 7 changes to pswiki, up to ID 687578481, timestamp 20180606092435. Lag is 217 seconds. Next ID is 687580509. 09:37:04 Posted 6 changes to pswiki, up to ID 687584827, timestamp 20180606093646. Lag is 18 seconds. Next ID is 687584957. 09:45:25 Posted 3 changes to pswiki, up to ID 687588950, timestamp 20180606094344. Lag is 101 seconds. Next ID is 687589974.
When the dispatcher finally got around to dispatching to the client again at each of the below times the dispatch lag was:
06:58:38 - 24 mins + 67 seconds =~ 25 mins 07:22:13 - 24 mins + 242 seconds =~ 28 mins 07:32:45 - 10 mins + 86 seconds =~ 11 mins 07:42:08 - 10 mins + 47 seconds =~ 11 mins 07:51:58 - 9 mins + 66 seconds =~ 10 mins 08:04:48 - 13 mins + 41 seconds =~14 minsSo pswiki for the hour had the average lag of ~15 mins.
The baseline lag can be seen as roughly 1-2 mins.On sites like enwiki the base lag that never gets dispatched past is more like ~250-600 seconds (even though 420 changes are dispatched during each pass.
Cc: Lucas_Werkmeister_WMDE, gerritbot, Aklapper, hoo, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, lisong, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
