| Marostegui added a comment. |
In T184812#4010371, @Lucas_Werkmeister_WMDE wrote:Okay… is there anything I can do to help move that task forward?
The reason I want to get caching into place soon is that we’ll roll out constraint checks to more users in the next weeks (starting on March 1st to all usernames beginning with “Z”, see T184069#4000941 for details), so we will start to see more constraint check requests. I think we can bear the first group (just “Z”) without caching, but after that we would probably have to pause the rollout if we can’t get caching working by then. (See also T188386: Delay rollout of constraint violation gadget to all users?.)
I don't think we should be rushing things here, we already had a close call on Monday.
We were pretty pretty pretty close to saturate the network interfaces of the databases.
Our logtash logged thousands of errors during the time the change was in production, and again, we were pretty close to completely saturate the servers, which could have ended up in a serious outage (throwing errors to 100% of the requests) and probably also overloading the master.
If we are going to keep rolling new things this fast, I would consider investigating and fixing T188505 a must. Why? Basically to be on the safe side, in case another bad deployment happens. It is better to kill queries than get all the servers overloaded and suffer a major outage for wikidata.
Cc: Lydia_Pintscher, Bawolff, greg, Ladsgroup, jcrespo, Marostegui, TerraCodes, Stashbot, Liuxinyu970226, Jonas, Aklapper, gerritbot, Lucas_Werkmeister_WMDE, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, HJiang-WMF, LawExplorer, Lewizho99, Maathavan, Agabi10, dpatrick, Luke081515, Wikidata-bugs, aude, GWicke, Stype_and_Co.-WMF, Jalexander, ArielGlenn, Parent5446, Anomie, Grunny, He7d3r, csteipp, Mbch331, Jay8g, Legoktm
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
