Thanks @jyapayne. I think we (i.e., you) are really close. The first character
of each line that exceeds the buffer was getting cut off (or maybe if the prior
line exceeded buffer?). Looks like we're off by one at the same spot. I believe:
buffer.add data[pos+1 ..< pos+bufSize]
Run
should be:
buffer.add data[pos ..< pos+bufSize] # Without '+1'
Run
Looks like pos already gets set to `last + 1` in the prior conditional
statement. What I'm not sure about is how this should affect the following
statement. Should that be `pos-1`?
I plan to play more with @cblakes code, but I'm curious to see this through,
out of curiosity.
Using your updated code, combined with my minor change, the script parses my
large VCF in just under `12min`, which is just a bit longer than the Groovy
code. Sounds like `popen` from @cblakes will speed that up by decompressing in
a second thread, but that Nim and Groovy are fairly similar unzipping and
parsing in a single thread. Or did I misunderstand something?
