[Wikidata-bugs] [Maniphest] [Commented On] T206535: wikidata weekly dumps take too long to complete

[ArielGlenn]

ArielGlenn added a comment.

In T206535#4675178, @hoo wrote:

...

Have you tested importing that via php (and/or anything else that uses the libzip2 compat stuff)?

script:

[ariel@bigtrouble ~]$ more catbz2file.php
<?php
/*
 * Uncompress a bzip2-compressed file and write it to stdout
 */
$filename = 'compress.bzip2://' . '/mnt/dumpsdata/temp/ariel/wikidata-20181015-all-BETA.ttl.bz2';
// $filename = 'compress.bzip2://' . '/home/ariel/wmf/dumps/lbzip2/commonswiki-20180520-stub-meta-current4.xml.bz2';
$betattl = fopen($filename, "r");
while (($line = fgets($betattl, 4096)) !== false) {
    echo $line;
}
fclose($betattl);

md5sum of original (gz) file:

ariel@dumpsdata1002:/data/otherdumps/wikibase/wikidatawiki/20181015$ zcat wikidata-20181015-all-BETA.ttl.gz | md5sum
6ae514b7b889f55c787e37bdf6cc72ed  -

md5sum of lbzip2-ed file:

ariel@snapshot1008:~$  /usr/bin/php7.0 catbz2file.php  | md5sum
6ae514b7b889f55c787e37bdf6cc72ed  -

TASK DETAIL

https://phabricator.wikimedia.org/T206535

EMAIL PREFERENCES

https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: hoo, Smalyshev, ArielGlenn, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, LawExplorer, gnosygnu, Wikidata-bugs, aude, Mbch331

_______________________________________________
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs