tstarling added a comment.
This proposal is effectively a dynamic rate limit except that instead of delivering an error message when it is exceeded, we will just hold the connection open, forcing the bot to wait. That's expensive in terms of server resources -- we'd rather have the client wait using only its own resources. A rate limit has a tunable parameter (the rate) which is not really knowable. Similarly, this proposal has a tunable parameter (the pool size) which is not really knowable. You have to tune the pool size down until the replag stops increasing, but then if the nature of the edits changes, or if the hardware changes, the optimal pool size will change. I suggested at T202107 <https://phabricator.wikimedia.org/T202107> that the best method for globally controlling replication lag would be with a PID controller <https://en.wikipedia.org/wiki/PID_controller>. A PID controller suppresses oscillation by having a memory of recent changes in the metric. The P (proportional) term is essentially as proposed at T240442 <https://phabricator.wikimedia.org/T240442> -- just back off proportionally as the lag increases. The problem with this is that it will settle into an equilibrium lag somewhere in the middle of the range. The I (integral) term addresses this by maintaining a rolling average and adjusting the control input until the average meets the desired value. This allows it to maintain approximately the same edit rate but with a lower average replication lag. The D (derivative) term causes the control input to be reduced more aggressively if the metric is rising quickly. My proposal is to use a PID controller to set the Retry-After header. Clients would be strongly encouraged to respect that header. We could have say maxlag=auto to opt in to this system. TASK DETAIL https://phabricator.wikimedia.org/T252091 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: tstarling Cc: tstarling, Joe, Dvorapa, daniel, Krinkle, Aklapper, Jakob_WMDE, Lydia_Pintscher, WMDE-leszek, darthmon_wmde, Addshore, Ladsgroup, DannyS712, Nandana, kostajh, Lahi, Gq86, GoranSMilovanovic, RazeSoldier, QZanden, LawExplorer, elukey, _jensen, rosalieper, D3r1ck01, Scott_WUaS, Jonas, Izno, SBisson, Perhelion, Wikidata-bugs, Base, aude, GWicke, Bawolff, jayvdb, fbstj, santhosh, Jdforrester-WMF, Mbch331, Rxy, Jay8g, Ltrlg, bd808, Legoktm
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
