Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-03-13T18:42:18Z] repool wdqs1004 & wdqs2001 now that data reload is completed T189548TASK DETAILhttps://phabricator.wikimedia.org/T189548EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To:
Gehel added a comment.
After experimenting a bit, I removed gzip from the pipeline. It looks like gzip is CPU bound (and not multi-threaded). Even with gzip -1, the transfer rate is slower than with no compression.TASK DETAILhttps://phabricator.wikimedia.org/T189548EMAIL
Gehel added a comment.
Data transfer done with:
wdqs1004 (receiving):
nc -l -p 9876 | gunzip | pv -b -r > wikidata.jnl
wdqs2001 (sending):
cat wikidata.jnl | gzip -9 | nc -w 3 wdqs1004.eqiad.wmnet 9876
The transfer is not encrypted (but does not contain any PII). Checking the result with
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-03-13T10:00:04Z] shuttind down blazegraph on wdqs2001 for data transfer to wdqs1004 - T189548TASK DETAILhttps://phabricator.wikimedia.org/T189548EMAIL
Gehel added a comment.
@Smalyshev yes, there is a way to copy the data between wdqs nodes, I'll take care of it and document it here. The new wdqs cluster is not yet done reloading, so I'll take the data from wdqs2001 (I prefer to shutdown a node in codfw than running on a single node in eqiad,
Smalyshev added a comment.
@Gehel is there any easy way to copy the .jnl file between machines? It's 413G but probably compressable so we'll be moving about 100-200G of data if we zip it up. But I am not sure how to do it. Or we reload from dump but then we'd have to wait till Thursday/Friday or
