Ladsgroup added a comment.
Okay, Let me give on overview on what has been worked on so far, this is
basically a summary of the previous comments and reshaped:
- Queries to term store are basically half of the load.
- Given that it's actually 1% of queries that actually reach the db after
being handled by memcached and APCu, I repeat, this is scary.
- At least half of our load is from client wikis but probably no more than
70% (getting the exact number is impossible)
- Finding out how much of queries are coming from the API appservers (like
ratio of open connections) can be done by DBAs I assume, I can't find any other
way to find the API/non-API context for queries.
- Also Special:EntityData doesn't matter in that regard because most of the
heavy lifting is to ES and not s8
- There's a trade-off, if we want to achieve resiliency (e.g. by separating
portions of queries to a dedicated node), it would decrease scalability and
flexibility of the infra, I recommend getting DBAs opinion on the decision too.
- Load should not be the only factor in building groups, the type of queries
and tables it queries is important, for example, recentchanges is only 1% of
the load (not number of queries, the total latency) on s8, it doesn't make
sense to have a dedicated node (per DC) for it but having them localized to one
node makes sense so the queries take advantage of MySQL caching (of course, the
node should not be only handling rc queries otherwise it would be a huge waste
of resource).
My personal preference in this is grouping term store due to the fact that
it's going to be cached better in MySQL and the load is not negligible and it's
pretty straightforward to implement (e.g. client/repo is not easy to implement
given that most of them hit lib/ instead or lots of them actually coming from
mediawiki core and we have no control over destination of queries originating
from there). My other idea about this is that in the mid-term planning (like in
5 years), all of term store can be moved to its dedicated section to
drastically increase the scalability of wikidata's database in matter of
read/write/storage, while such thing can't be achieved via repo/client split.
OTOH, the resilience that repo/client can bring won't be achieved because
client and repo queries won't be separated and issues originating from one
would affect the other OTOOH (I have three hands), most of the load on s8 comes
from mw core and lib code (and not repo/client code) thus bugs and issues in
lib will bring down both anyway even if separate them.
TASK DETAIL
https://phabricator.wikimedia.org/T246415
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ladsgroup
Cc: Marostegui, Ladsgroup, WMDE-leszek, Aklapper, Addshore, Alter-paule,
Beast1978, Un1tY, Akuckartz, Hook696, Iflorez, darthmon_wmde, Kent7301,
alaa_wmde, joker88john, CucyNoiD, Nandana, jijiki, Klaas_Z4us_V, Gaboe420,
Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Pablo-WMDE,
GoranSMilovanovic, QZanden, LawExplorer, Lewizho99, Maathavan, elukey, _jensen,
rosalieper, Scott_WUaS, Jonas, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331,
Jay8g
_______________________________________________
Wikidata-bugs mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
