bking added a comment.
A few notes on this process:
- I used zstd compression <https://facebook.github.io/zstd/> to compress
the JNL file, as it supposedly offers the best speed. I used `zstd -T0 -19
wikidata.jnl` as my compression command (all cores, maximum compression), using
`wdqs1016` as the host. Despite having 32 cores, I never saw load average go
past 16 during the compression process. The compression process took ~12 hours.
- I uploaded to Cloudflare R2 object storage using @Addshore 's rclone
command as described here
<https://addshore.com/2023/08/wikidata-query-service-blazegraph-jnl-file-on-cloudflare-r2-and-internet-archive/>
. The upload process (with the command optimizations in the linked post) took
about an hour.
- I'm downloading the file using axel
<https://github.com/axel-download-accelerator/axel> over my 1Gbps post, using
16 concurrent connections: `axel -a -n 16`, getting about ~320Mbps transfer
speed.
TASK DETAIL
https://phabricator.wikimedia.org/T347605
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: bking
Cc: Addshore, dr0ptp4kt, Aklapper, bking, Danny_Benjafield_WMDE, Astuthiodit_1,
AWesterinen, BTullis, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE,
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE,
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper,
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984,
Manybubbles, Mbch331
_______________________________________________
Wikidata-bugs mailing list -- [email protected]
To unsubscribe send an email to [email protected]
