Igorkim78 added a comment.
Wdqs1006 reports 574.6GiB are reserved for the journal and 544.3GiB are actually used (~5% of space unused). While Wdqs1005 reports 1037.7GiB are reserved and only 543.5 are actully used (~47% of space unused). Most of the %FileWaste or reserved for 8K allocators, but %SlotWaste is also higher than usual for 4k (10 times higher than usual), 2k, 64 (3 times), 320 and 768 allocators (2 times). Slots allocated using 8k allocators are similar on both servers (less than 5% difference) about 5,179M vs 5,431M and only ~1% of them remain in use ~63M. This happens due to updates, for each update parts of the indices related to the changing data have to be copied to a new allocator with changes applied, then the old allocator might be marked as unused and then reused for the later updates after all connections which refer to the commit point linked to the mentioned allocators are closed. But if a commit point could not be released, the allocators are also remain locked. Analyzing Graphana reports, I assume most of the allocators where consumed gradually from Nov 1, 6:00 to Nov 4, 18:00. Given all the above, the conclusion is that something (most probably some intentionally or unintentionally unclosed connection) was blocking releasing allocators for 3.5 days preventing their reuse, thus updates had to allocate new allocators, then the commit point was released and the locked allocators are also released, but they could not be removed from file, just increasing sparse space. TASK DETAIL https://phabricator.wikimedia.org/T238232 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Igorkim78 Cc: Mathew.onipe, Igorkim78, Gehel, Aklapper, darthmon_wmde, DannyS712, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
