daniel added a comment.
In T138208#7727264 <https://phabricator.wikimedia.org/T138208#7727264>, @Ladsgroup wrote: > Possibly but also keeping the connection open? Maybe it needs to buffer, close the connection and then compress given that it's cpu intensive and slow? WikiExporter writes each chunk of xml to an DumpOutput. In the above case, that would be a DumpBZip2Output, which is a DumpPipeOutput, which uses proc_open to start bzip2 and then writes each chunk to the child process's stdin. The output is far too big to buffer in memory. Writing to disk uncompressed may be an option, bout would require an order of magnitude more disk space. And the wrapper scripts would need to be changed significantly, I suppose. TASK DETAIL https://phabricator.wikimedia.org/T138208 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: daniel Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, ArielGlenn, jcrespo, Zppix, karapayneWMDE, Invadibot, maantietaja, jannee_e, Akuckartz, holger.knust, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, aude, Mbch331
_______________________________________________ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
