Addshore added a subscriber: jcrespo.
Addshore added a comment.
Spent a lot of time chatting with @jcrespo today.
After some experimenting it seems that the main issue that we have been
seeing was probably caused by disk reads due to data not being in the db cache
yet.
This slowness of disk reads was also added to as these db tables are
compressed (at least on db1126).
I got to Q6 million for reads from clients 30 mins or so ago and everything
seems stable.
I turned off the batch write migration this morning but turned it back on 10
mins ago (batch size 100) and everything still seems to be indicating that it
is okay.
The code snippet for warming the cache of db1126 is below for the range 2-4
million, and can be run from the mwmaint servers.
for i in {2000000..4000000..100000}; do echo "$i..."; time sql wikidatawiki
--host db1126 -- -e "SELECT wbit_item_id as id, wby_name as type,
wbxl_language as language, wbx_text as text FROM wbt_item_terms LEFT JOIN
wbt_term_in_lang ON wbit_term_in_lang_id = wbtl_id LEFT JOIN wbt_type ON
wbtl_type_id = wby_id LEFT JOIN wbt_text_in_lang ON wbtl_text_in_lang_id =
wbxl_id LEFT JOIN wbt_text ON wbxl_text_id = wbx_id WHERE wbit_item_id BETWEEN
$i AND ($i + 100000)" > /dev/null; done
This cache warmer should be performed twice on each range shortly before
turning on reads from the new store for that range.
The cache efficiency can be seen in the below snapshot where:
a) cache warm 2-4 million pass 1
b) cache warm 2-4 million pass 2
c) cache warm 4-6 million pass 1
d) cache warm 4-6 million pass 2
F31647226: image.png <https://phabricator.wikimedia.org/F31647226>
This can also be seen in the disk IO
F31647231: image.png <https://phabricator.wikimedia.org/F31647231>
Although I have started an initial cache warm for Q6-8 million I probably
won't bump the read level higher today.
TASK DETAIL
https://phabricator.wikimedia.org/T219123
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ladsgroup, Addshore
Cc: jcrespo, WMDE-leszek, Ladsgroup, Addshore, Jdforrester-WMF, ArielGlenn,
Aklapper, alaa_wmde, Hazizibinmahdi, Beast1978, Un1tY, Hook696, Daryl-TTMG,
RomaAmorRoma, E.S.A-Sheild, Iflorez, darthmon_wmde, Meekrab2012, joker88john,
CucyNoiD, Nandana, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22,
Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Af420, Darkminds3113, Bsandipan,
Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden,
LawExplorer, WSH1906, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS,
Jonas, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331
_______________________________________________
Wikidata-bugs mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
