On 10/17/17 4:33 AM, ikod wrote:
Hello, Steve
On Friday, 13 October 2017 at 22:22:54 UTC, Steven Schveighoffer wrote:
On 10/13/17 6:18 PM, ikod wrote:
On Friday, 13 October 2017 at 19:17:54 UTC, Steven Schveighoffer wrote:
Eventually, something like this will be possible with jsoniopipe (I
need to update and release this too, it's probably broken with some
of the changes I just put into iopipe). Hopefully combined with some
sort of networking library you could process a JSON stream without
reading the whole thing into memory.
This can be done with requests. You can ask not to load whole content
in memory, but instead produce input range, which will continue to
load data from server when you will be ready to consume:
auto rq = Request();
rq.useStreaming = true;
auto rs = rq.get("http://httpbin.org/image/jpeg";);
auto stream = rs.receiveAsRange();
while(!stream.empty) {
// stream.front contain next data portion
writefln("Received %d bytes, total received %d from document
legth %d", stream.front.length, rq.contentReceived, rq.contentLength);
stream.popFront; // continue to load from server
}
Very nice, I will add a component to iopipe that converts a
"chunk-like" range like this into an iopipe source, as this is going
to be needed to interface with existing libraries. I still will want
to skip the middle man buffer at some point though :)
Thanks!
Just in order to have complete picture here - getContent returns not
just ubyte[], but more rich structure (which can be convered to ubyte[]
if needed). Basically it is an immutable(immutable(ubyte)[]) and almost
all data there are just data received from network without any data copy.
Right, iopipe can use it just fine, without copying, as all arrays are
also iopipes. In that case, it skips allocating a buffer, because there
is no need.
However, I prefer the need to avoid allocating the whole thing in
memory, which is why I would prefer the range interface. However, in
this case, iopipe needs to copy each chunk to its own buffer.
In terms of the most useful/least copying, direct access to the stream
itself would be the best, which is why I said "skip the middle man". I
feel like this won't be possible directly with requests and iopipe,
because you need buffering to deal with parsing the headers. I think
it's probably going to be a system built on top of iopipe, using its
buffers, that would be the most optimal.
There are more details and docs on
https://github.com/ikod/nbuff/blob/master/source/nbuff/buffer.d. Main
goal behind Buffer is to minimize data movement, but it also support
many range properties, as long as some internal optimized methods.
I will take a look when I get a chance, thanks.
-Steve
