It looks like `nim-faststreams` is just to bridge the API gap between mmap IO
and streams interfaces which is nice and all, but won't help for `startProcess`
or other `popen`-like contexts where in this discussion streams slowness was a
problem. { @mratsim didn't say it would, exactly, but I thought I should
clarify (unless I'm wrong). }
More importantly, perhaps to @markebbert's surprise, it turns out that the
bioinformatics guys are being smarter than `pigz` with this `bgzip` tool which
supports parallel decompression. That very same example file that's been my
running example since @jyapayne mentioned it is in fact `bgzip` d. It can
decompress in only 11.1 seconds on the same machine with 4 threads. (I got it
down to 3.5 seconds with over 25 threads but that same Zstd example gets down
to 2.2s using similarly many threads). So, an `mSlices` \+ `popen("bgzip -d
--threads 4 <
ALL.chrX.phase3_shapeit2_mvncall_integrated_v1b.20130502.genotypes.vcf.gz")`
version should be able to go in roughly 1 minute for that file.
As @brentp surely knows, his `hts-nim` library actually supports a
`set_threads` and `read_line` that can deliver that, too. It's entirely
possible @markebbert could have been getting that much faster decompress time
all along (with python and groovy as well via a popen analogue). Or maybe his
`.gz` is not from `bgzip` and he couldn't. The way for him to tell is to run
`file` on the `.gz` and see if it has "Blocked GNU Zip Format (BGZF; gzip
compatible), block length" instead of simply "gzip compressed data".
Of course, the compression ratio will still be 12x worse than Zstd. Besides the
obvious space waste, rather than a tiny 45-75 MB/s you would need backing IO of
172..542 MB/s (depending on 4 or 25 threads). On the high end many thread case,
you're looking at either a fast SSD or a RAID array of spinning disks to feed
that analysis pipeline (assuming zero cost parsing, anyway, which I know his
example calculation did not have, but that assumption simplifies since he
didn't really show a full calculation).
