Hoo man has uploaded a new change for review.
https://gerrit.wikimedia.org/r/249981
Change subject: Also published bzip2 compressed Wikidata TTL dumps
......................................................................
Also published bzip2 compressed Wikidata TTL dumps
As there seems to be a demand for this.
Change-Id: I6210e95ed879fe604e353eea933498b8ad8396c1
---
M modules/snapshot/files/dumpwikidatattl.sh
1 file changed, 8 insertions(+), 2 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/81/249981/1
diff --git a/modules/snapshot/files/dumpwikidatattl.sh
b/modules/snapshot/files/dumpwikidatattl.sh
index d1cf85e..b15c30c 100644
--- a/modules/snapshot/files/dumpwikidatattl.sh
+++ b/modules/snapshot/files/dumpwikidatattl.sh
@@ -7,7 +7,8 @@
. /usr/local/bin/wikidatadumps-shared.sh
filename=wikidata-$today-all-BETA
-targetFile=$targetDir/$filename.ttl.gz
+targetFileGzip=$targetDir/$filename.ttl.gz
+targetFileBzip2=$targetDir/$filename.ttl.bz2
i=0
shards=4
@@ -21,11 +22,16 @@
i=0
while [ $i -lt $shards ]; do
- cat $tempDir/wikidataTTL.$i.gz >> $targetFile
+ cat $tempDir/wikidataTTL.$i.gz >> $tempDir/wikidataTtl.gz
rm $tempDir/wikidataTTL.$i.gz
let i++
done
+mv $tempDir/wikidataTtl.gz $targetFileGzip
+
+gzip -dc $targetFileGzip | pbzip2 -p3 -c > $tempDir/wikidataTtl.bz2
+mv $tempDir/wikidataTtl.bz2 $targetFileBzip2
+
pruneOldDirectories
pruneOldLogs
runDcat
--
To view, visit https://gerrit.wikimedia.org/r/249981
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I6210e95ed879fe604e353eea933498b8ad8396c1
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Hoo man <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
