On Friday, 13 October 2017 at 19:17:54 UTC, Steven Schveighoffer
wrote:
On 10/13/17 2:47 PM, Andrew Edwards wrote:
A bit of advice, please. I'm trying to parse a gzipped JSON
file retrieved from the internet. The following naive
implementation accomplishes the task:
auto url =
"http://api.syosetu.com/novelapi/api/?out=json&lim=500&gzip=5";;
getContent(url)
.data
.unzip
.runEncoded!((input) {
ubyte[] content;
foreach (line; input.byLineRange!true) {
content ~= cast(ubyte[])line;
}
auto json = (cast(string)content).parseJSON;
input is an iopipe of char, wchar, or dchar. There is no need
to cast it around.
In this particular case, all three types (char[], wchar[], and
dchar[]) are being returned at different points in the loop. I
don't know of any other way to generate a unified buffer than
casting it to ubyte[].
Also, there is no need to split it by line, json doesn't care.
I thought as much but my mind was not open enough to see the
solution.
Note also that getContent returns a complete body, but unzip
may not be so forgiving. But there definitely isn't a reason to
create your own buffer here.
this should work (something like this really should be in
iopipe):
while(input.extend(0) != 0) {} // get data until EOF
This!!! This is what I was looking for. Thank you. I incorrectly
assumed that if I didn't process the content of input.window, it
would be overwritten on each .extend() so my implementation was:
ubyte[] json;
while(input.extend(0) != 0) {
json ~= input.window;
}
This didn't work because it invalidated the Unicode data so I
ended up splitting by line instead.
Sure enough, this is trivial once one knows how to use it
correctly, but I think it would be better to put this in the
library as extendAll().
And then:
Eventually, something like this will be possible with
jsoniopipe (I need to update and release this too, it's
probably broken with some of the changes I just put into
iopipe). Hopefully combined with some sort of networking
library you could process a JSON stream without reading the
whole thing into memory.
That would be awesome. Again, thank you very much.
Andrew
