ArielGlenn added a comment.
Ran
php /srv/mediawiki/multiversion/MWScript.php
extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki commonswiki
--batch-size 500 --format nt --flavor full-dump --entity-type mediainfo
--no-cache --dbgroupdefault dump --ignore-missing --first-page-id 78846320
--last-page-id 79046320 --shard 0 --sharding-factor 1
2>/var/lib/dumpsgen/mediainfo-log-small-shard-oom.txt | gzip >
/mnt/dumpsdata/temp/dumpsgen/mediainfo-dumps-test-nt-one-shard-small-oom.gz
which should cover the page range where we had the oom; it ran to completion
fine. I guess that there is some small memory leak that must accumulate over
batches, which is what did us in earlier. As long as we limit runs to some
reasonable number of pages each time, we should be fine.
TASK DETAIL
https://phabricator.wikimedia.org/T221917
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: ArielGlenn
Cc: Cparle, Abit, Gehel, jleedev, hoo, ArielGlenn, WMDE-leszek, Poyekhali,
Steinsplitter, Aklapper, Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt,
Addshore, Tpt, Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, Hook696, Daryl-TTMG,
RomaAmorRoma, 0010318400, E.S.A-Sheild, darthmon_wmde, Meekrab2012,
joker88john, CucyNoiD, Nandana, NebulousIris, JKSTNK, Gaboe420, Versusxo,
Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86,
Af420, E1presidente, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis,
GoranSMilovanovic, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden,
EBjune, Tramullas, Acer, merbst, LawExplorer, WSH1906, Lewizho99, Maathavan,
Silverfish, _jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer,
Susannaanas, Ixocactus, Wong128hk, gnosygnu, Jane023, jkroll, Wikidata-bugs,
Jdouglas, Base, matthiasmullie, aude, Tobias1984, El_Grafo, Dinoguy1000,
Manybubbles, Ricordisamoa, Wesalius, Fabrice_Florin, Raymond, Jdforrester-WMF,
Mbch331, Keegan
_______________________________________________
Wikidata-bugs mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
