Because various Zstd ratios are all so large, it helps in many practical 
circumstances more than choice of programming language (which often plays in 
the 2x-5x range). Continuing with that one data file example, with just 4 cores 
the output rate is 7.3 GB/s. On an otherwise idle 16 core system, I get 
decompress times of 2.26 sec => output rate of 11.93 GB/s (1 TB in 83 seconds, 
1 PB in a day).

These numbers are really much faster than almost all IO systems these days. 
(Not "all", but probably "all except crazy fat RAIDs of 4 or more NVMe's").

To keep that data pipeline fed, you only need 165 MB/2.26s =~ 73 MB/s which is 
also well within reach of most IO systems. So, getting that 7-12 GB/s vs gzip's 
paltry 345 MB/s decompress rate is quite achievable and 20X..34X speed boost vs 
gzip is compelling.

Assuming that compression ratio of 163:1 holds up for other VCF files, that 
petabyte of VCF output would only take up like 6 TB disk which a lot of people 
have lying around these days. And people like markebbert are apparently patient 
enough to wait for 7 hours. So, in some sense, petabyte-scale home computing 
has been around for half a decade now "on the down low for the somewhat 
patient".

I'm a little surprised it isn't more well known/widely discussed. Not 
everything compresses like gangbusters the way this VCF data does, though.

Reply via email to