On 10/13/17 2:47 PM, Andrew Edwards wrote:
A bit of advice, please. I'm trying to parse a gzipped JSON file retrieved from the internet. The following naive implementation accomplishes the task:

    auto url = "http://api.syosetu.com/novelapi/api/?out=json&lim=500&gzip=5";;
     getContent(url)
         .data
         .unzip
         .runEncoded!((input) {
             ubyte[] content;
             foreach (line; input.byLineRange!true) {
                 content ~= cast(ubyte[])line;
             }
             auto json = (cast(string)content).parseJSON;

input is an iopipe of char, wchar, or dchar. There is no need to cast it around.

Also, there is no need to split it by line, json doesn't care.

Note also that getContent returns a complete body, but unzip may not be so forgiving. But there definitely isn't a reason to create your own buffer here.

this should work (something like this really should be in iopipe):

while(input.extend(0) != 0) {} // get data until EOF

And then:
auto json = input.window.parseJSON;

             foreach (size_t ndx, record; json) {
                 if (ndx == 0) continue;
                 auto title = json[ndx]["title"].str;
                 auto author = json[ndx]["writer"].str;
                 writefln("title: %s", title);
                 writefln("author: %s\n", author);
             }
         });

However, I'm sure there is a much better way to accomplish this. Is there any way to accomplish something akin to:

    auto url = "http://api.syosetu.com/novelapi/api/?out=json&lim=500&gzip=5";;
     getContent(url)
         .data
         .unzip
         .runEncoded!((input) {
             foreach (record; input.data.parseJSON[1 .. $]) {
                 // use or update record as desired
             }
         });

Eventually, something like this will be possible with jsoniopipe (I need to update and release this too, it's probably broken with some of the changes I just put into iopipe). Hopefully combined with some sort of networking library you could process a JSON stream without reading the whole thing into memory.

Right now, it works just like std.json.parseJSON: it parses an entire JSON message into a DOM form.

-Steve

Reply via email to