Gehel created this task. Gehel added projects: serviceops, Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. Restricted Application added a project: Wikidata.
TASK DESCRIPTION As an operator of WDQS, I want to ensure that the Updater isn't overloading dependent services by limiting max concurrent requests. Given the recent incident where the WDQS Flink based streaming updater was blocked as it seemed to generate too much traffic, we want to enforce appropriate limits on the query parallelism it can generate. The implementation is already in place, this is just a matter of finding the appropriate configuration. This new updater is a lot more efficient than the current one (which is the whole point), so it can potentially generate a lot more load. As a distributed application, this updater has concurrency limits on each node (currently, 12 nodes x 6 thread = 72 concurrent requests max). Note that this max number is never reached, each nodes does other things than querying Wikidata. In normal operations, the number of requests is limited by the edit rate on Wikidata, and whatever limit we set must be sufficient to support that edit rate. During initial data load, the backlog of edits is consumed as fast as possible, limited by whatever concurrency we set. This is where having a reasonable limit is necessary. The current updater duplicates requests to wikidata for each node (18 nodes at the moment). The new updater centralises this, so we can expect a 18x reduction in queries during normal operation once the new updater is pushed to production. AC: - parallelism limits are agreed on with Service Ops and configured TASK DETAIL https://phabricator.wikimedia.org/T275133 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Gehel, Aklapper, MPhamWMF, wkandek, JMeybohm, CBogen, Akuckartz, Nandana, Namenlos314, jijiki, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Dzahn
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
