bking added a comment.
A few notes on this process: - I used zstd compression <https://facebook.github.io/zstd/> to compress the JNL file, as it supposedly offers the best speed. I used `zstd -T0 -19 wikidata.jnl` as my compression command (all cores, maximum compression), using `wdqs1016` as the host. Despite having 32 cores, I never saw load average go past 16 during the compression process. The compression process took ~12 hours. - I uploaded to Cloudflare R2 object storage using @Addshore 's rclone command as described here <https://addshore.com/2023/08/wikidata-query-service-blazegraph-jnl-file-on-cloudflare-r2-and-internet-archive/> . The upload process (with the command optimizations in the linked post) took about an hour. - I'm downloading the file using axel <https://github.com/axel-download-accelerator/axel> over my 1Gbps post, using 16 concurrent connections: `axel -a -n 16`, getting about ~320Mbps transfer speed. TASK DETAIL https://phabricator.wikimedia.org/T347605 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Addshore, dr0ptp4kt, Aklapper, bking, Danny_Benjafield_WMDE, Astuthiodit_1, AWesterinen, BTullis, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
_______________________________________________ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org