ArielGlenn added a comment.
Timing test results:
With bzip2:
dumpsgen@snapshot1006:/mnt/dumpsdata/temp/dumpsgen$ time /usr/bin/php7.2
/srv/mediawiki/multiversion/MWScript.php dumpTextPass.php --wiki=wikidatawiki
--stub=gzip:/mnt/dumpsdata/temp/dumpsgen/wikidatawiki-20190301-stub-meta-history26.xml-p29211328p29248936.gz
--prefetch=7zip:/mnt/dumpsdata/temp/dumpsgen/wikidatawiki-20190201-pages-meta-history26.xml-p29211328p29248936.7z
--report=1000 --spawn=/usr/bin/php7.2
--output=bzip2:/mnt/dumpsdata/temp/dumpsgen/testoutput/wikidatawiki-20190301-pages-meta-history26.xml-p29211328p29248936.bz2
--full
...
real 943m50.096s
user 948m31.120s
sys 6m15.644s
With lbzip2:
dumpsgen@snapshot1005:/mnt/dumpsdata/temp/dumpsgen$ time /usr/bin/php7.2
/srv/mediawiki/multiversion/MWScript.php dumpTextPass.php --wiki=wikidatawiki
--stub=gzip:/mnt/dumpsdata/temp/dumpsgen/wikidatawiki-20190301-stub-meta-history26.xml-p29211328p29248936.gz
--prefetch=7zip:/mnt/dumpsdata/temp/dumpsgen/wikidatawiki-20190201-pages-meta-history26.xml-p29211328p29248936.7z
--report=1000 --spawn=/usr/bin/php7.2
--output=lbzip2:/mnt/dumpsdata/temp/dumpsgen/testoutput/wikidatawiki-20190301-pages-meta-history26.xml-p29211328p29248936-new.bz2
--full
...
real 130m21.197s
user 180m22.988s
sys 7m5.904s
9 hours 45 min for bzip2, vs 2 hours 10 min for lbzip2 with one core, which
is how we'd be running it in practice. That's for the biggest of the produced
files, most are much smaller. I'd like to move ahead with this change.
TASK DETAIL
https://phabricator.wikimedia.org/T214293
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: ArielGlenn
Cc: gerritbot, hoo, ArielGlenn, alaa_wmde, CucyNoiD, Nandana, NebulousIris,
Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30,
Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic,
Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer,
Vali.matei, Lewizho99, Maathavan, _jensen, rosalieper, Volker_E, gnosygnu,
Wikidata-bugs, aude, GWicke, Dinoguy1000, Mbch331, Jay8g
_______________________________________________
Wikidata-bugs mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
