On Monday, 29 June 2020 at 11:58:10 UTC, Steven Schveighoffer
wrote:
On 6/28/20 8:37 PM, aberba wrote:
On Sunday, 28 June 2020 at 20:26:43 UTC, JN wrote:
What's iopipe and what does it do? How does it compare with
std.process?
I my line of words, its what you'd use to stream large files
and do processing on it. Like CSV, video??, Json, and the
like. Its high performance cus it optimized to haven great
performance even for large files. Designed to let you chain
the processing pipeline using reusable functions...and you
can compose one yourself.
Yes, the emphasis is on being able to handle large sets of data
similarly to how you would handle an array, but not having to
load the whole thing into an array. It uses buffering to
present an array-like interface. The only difference between it
and arrays is you have to tell it how much data is important to
you at the moment.
There is also a focus on composability -- I want you to be able
to build any type of harness to take what you have and turn it
into something you can feed into a parser/handler.
Thanks for the expansion. I could really use this for an article.
Added to my to-do list.
Take for example, jsoniopipe [1], which is a library that uses
iopipe to parse JSON. You can feed it any source -- a string, a
zipped file, a network socket, and all you have to do is
compose the pieces to get it to be a text iopipe (an iopipe
where the window type is a string type). Then the json library
will parse it and provide you with a stream of JSON data.
Because everything is available to the compiler, it can
optimize/inline across all the layers, and provide the best
performance possible (see my presentation at dconf 2017 [2] for
how this works out, also linked by Clarice).
Will looking into that library more.
and all you have to do is compose the pieces to get it to be a
text iopipe (an iopipe where the window type is a string type).
No sure I understand this statement.
I'm curious myself how it differs from the NodeJs Stream API.
Would you say iopipe is a Streaming API by definition?
I'm not familiar with NodeJs in general so I can't comment on
that. I would say iopipe is focused on making streaming more
like dealing with arrays, with as high performance as possible.
One thing I've always disliked about most streaming libraries
is that they hide the buffer as an implementation detail, but
it should be the focus.
How well does iopipe work in a multi-threaded environment?
-Steve
[1] https://code.dlang.org/packages/jsoniopipe
[2] https://www.youtube.com/watch?v=un-bZdyumog
I'll probably re-watch it before writing the article.
