On 10/13/17 6:18 PM, ikod wrote:
On Friday, 13 October 2017 at 19:17:54 UTC, Steven Schveighoffer wrote:
Eventually, something like this will be possible with jsoniopipe (I
need to update and release this too, it's probably broken with some of
the changes I just put into iopipe). Hopefully combined with some sort
of networking library you could process a JSON stream without reading
the whole thing into memory.
This can be done with requests. You can ask not to load whole content in
memory, but instead produce input range, which will continue to load
data from server when you will be ready to consume:
auto rq = Request();
rq.useStreaming = true;
auto rs = rq.get("http://httpbin.org/image/jpeg";);
auto stream = rs.receiveAsRange();
while(!stream.empty) {
// stream.front contain next data portion
writefln("Received %d bytes, total received %d from document
legth %d", stream.front.length, rq.contentReceived, rq.contentLength);
stream.popFront; // continue to load from server
}
Very nice, I will add a component to iopipe that converts a "chunk-like"
range like this into an iopipe source, as this is going to be needed to
interface with existing libraries. I still will want to skip the middle
man buffer at some point though :)
Thanks!
-Steve
