| daniel created this task. daniel added projects: Wikidata, MediaWiki-extensions-WikibaseRepository. Herald added a subscriber: Aklapper. |
The dispatchChanges script currently has a --dispatch-interval setting which dictates how often a client wiki is allowed get a batch of changes. The maximum throughput is $batch-size per $dispatch-interval, independently of the number of dispatcher processes running.
The intent behind --dispatch-interval is to give batches some time to "fill up" - dispatching to the same wiki too often would lead to lots of small batches, leading to increased overhead and making client side optimizations (like change coalescing) less effective.
The solution could be a --batch-grace with the following semantics: if less than a full batch of changes is ready to be dispatched, only send them if we didn't send any changes to that client for at least $batchGrace seconds. If the batch is full, always send it.
This logic would still cover the intent of --dispatch-interval, but would avoid throttling the throughput while trying to catch up on a massive backlog of changes.
Cc: Aklapper, daniel, GoranSMilovanovic, QZanden, Wikidata-bugs, aude, Mbch331
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
