Re: Processing a gzipped csv-file by line-by-line

2017-05-12 Thread Steven Schveighoffer via Digitalmars-d-learn
On 5/11/17 8:18 PM, H. S. Teoh via Digitalmars-d-learn wrote: On Wed, May 10, 2017 at 11:40:08PM +, Jesse Phillips via Digitalmars-d-learn wrote: If you can get the zip to decompress into a range of dchar then std.csv will work with it. It is by far not the fastest, but much speed is

Re: Processing a gzipped csv-file by line-by-line

2017-05-12 Thread Steven Schveighoffer via Digitalmars-d-learn
On 5/10/17 7:17 PM, Nicholas Wilson wrote: On Wednesday, 10 May 2017 at 22:20:52 UTC, Nordlöw wrote: What's fastest way to on-the-fly-decompress and process a gzipped csv-fil line by line? Is it possible to combine http://dlang.org/phobos/std_zlib.html with some stream variant of

Re: Processing a gzipped csv-file by line-by-line

2017-05-11 Thread Laeeth Isharc via Digitalmars-d-learn
On Friday, 12 May 2017 at 00:18:47 UTC, H. S. Teoh wrote: On Wed, May 10, 2017 at 11:40:08PM +, Jesse Phillips via Digitalmars-d-learn wrote: [...] H.S. Teoh mentioned fastcsv but requires all the data to be in memory. Or you could use std.mmfile. But if it's decompressed data, then it

Re: Processing a gzipped csv-file by line-by-line

2017-05-11 Thread H. S. Teoh via Digitalmars-d-learn
On Wed, May 10, 2017 at 11:40:08PM +, Jesse Phillips via Digitalmars-d-learn wrote: [...] > H.S. Teoh mentioned fastcsv but requires all the data to be in memory. Or you could use std.mmfile. But if it's decompressed data, then it would still need to be small enough to fit in memory. Well,

Re: Processing a gzipped csv-file by line-by-line

2017-05-11 Thread Jon Degenhardt via Digitalmars-d-learn
On Wednesday, 10 May 2017 at 22:20:52 UTC, Nordlöw wrote: What's fastest way to on-the-fly-decompress and process a gzipped csv-fil line by line? Is it possible to combine http://dlang.org/phobos/std_zlib.html with some stream variant of File(path).byLineFast ? I was curious what

Re: Processing a gzipped csv-file by line-by-line

2017-05-10 Thread Jesse Phillips via Digitalmars-d-learn
On Wednesday, 10 May 2017 at 22:20:52 UTC, Nordlöw wrote: What's fastest way to on-the-fly-decompress and process a gzipped csv-fil line by line? Is it possible to combine http://dlang.org/phobos/std_zlib.html with some stream variant of File(path).byLineFast ? You can't really parse a

Re: Processing a gzipped csv-file by line-by-line

2017-05-10 Thread Seb via Digitalmars-d-learn
On Wednesday, 10 May 2017 at 23:19:15 UTC, H. S. Teoh wrote: Also, if you need to parse lots of CSV data very fast, you might be interested in this: https://github.com/quickfur/fastcsv T Or asdf: https://github.com/tamediadigital/asdf

Re: Processing a gzipped csv-file by line-by-line

2017-05-10 Thread H. S. Teoh via Digitalmars-d-learn
On Wed, May 10, 2017 at 11:17:44PM +, Nicholas Wilson via Digitalmars-d-learn wrote: > On Wednesday, 10 May 2017 at 22:20:52 UTC, Nordlöw wrote: > > What's fastest way to on-the-fly-decompress and process a gzipped > > csv-fil line by line? > > > > Is it possible to combine > > > >

Re: Processing a gzipped csv-file by line-by-line

2017-05-10 Thread Nicholas Wilson via Digitalmars-d-learn
On Wednesday, 10 May 2017 at 22:20:52 UTC, Nordlöw wrote: What's fastest way to on-the-fly-decompress and process a gzipped csv-fil line by line? Is it possible to combine http://dlang.org/phobos/std_zlib.html with some stream variant of File(path).byLineFast ? I suggest you take a look

Re: Processing a gzipped csv-file by line-by-line

2017-05-10 Thread ketmar via Digitalmars-d-learn
Nordlöw wrote: What's fastest way to on-the-fly-decompress and process a gzipped csv-fil line by line? Is it possible to combine http://dlang.org/phobos/std_zlib.html with some stream variant of File(path).byLineFast ? iv.vfs[0] can do that (transparently decompress gzip files, and

Processing a gzipped csv-file by line-by-line

2017-05-10 Thread Nordlöw via Digitalmars-d-learn
What's fastest way to on-the-fly-decompress and process a gzipped csv-fil line by line? Is it possible to combine http://dlang.org/phobos/std_zlib.html with some stream variant of File(path).byLineFast ?