On 6/12/18 1:51 AM, DigitalDesigns wrote:
Could you explain some benefits specific to this implementation and a
bit of the functional aspects for a proper overview of it's capabilities
and why I should chose this method over others?
The things that I think make this approach better are:
1. Direct buffer access
Direct buffer access I have found is one of those ideas that doesn't
seem like it's that impressive until you start using it. Many times,
buffering is used in a generic library for optimization (basically
amortizing reads) and is an implementation detail hidden from your view.
Think of how FILE * keeps a large buffer of data inside itself, but only
gives you access to one character at a time.
This forces you to create your *own* buffering scheme on top of that.
What a waste! Iopipe allows you to use buffering for your purposes on
top of the benefits of amortization. It's my belief that this is why
iopipe's byline feature is 2x faster than Phobos'.
2. Using templates to their fullest
Iopipes are all templated on the buffer or iopipe underneath it. This
makes tings easily swappable. It's really cool to be able to take your
JSON or XML parser, and hook it onto an in-memory string in one line,
and then hook it onto a socket, and everything is optimized for that
situation. It takes the fun and flexibility of range programming and
brings it to i/o.
This is why iopipe's byline handles all forms of UTF, compared to Phobos
which only handles UTF8.
For example, I handle all forms of UTF with iopipe, with a decent set of
utilities. Here is a complete program using iopipe that converts any
type of UTF into another type, optimized for the specific situation:
https://github.com/schveiguy/iopipe/blob/master/examples/convert/convert.d
3. Compiler optimization for everything
All parts of iopipe, except for the low-level reads and writes (which
ironically are not really part of iopipe) are visible to the compiler
for inlining and optimization. I'm leveraging the power of the decades
of optimization experience that the compiler can provide. This makes it
easy to write code that performs well.
An anecdote: For my talk on iopipe in 2017
(http://dconf.org/2017/talks/schveighoffer.html) I wanted to have a live
demo showing the performance power. I literally was still working on it
2 or 3 days before, while at dconf. I had already written a JSON parser,
which was part of my presentation, but when I was showing it to another
D user (Daniel Murphy), I couldn't really answer the question "how does
it perform?". So he gave me a challenge -- do pretty printing on a JSON
file. Simple enough, with the parser I had already written, took me
about 1 hour to write it. It performed poorly compared to what we would
have expected, but tweaking a few things (almost all were due to using
some algorithms incorrectly), I got it to go faster than RapidJson in
certain use cases, and reasonably close in others. And I did nothing in
terms of lookup tables or using any special instructions. All in all, it
was probably 2 hours of work, and the code is beautiful IMO!
https://github.com/schveiguy/jsoniopipe/blob/master/examples/formatjson/formatjson.d
I think anyone who is doing parsing should have a look at iopipe, it not
only may make your code much simpler and easier to read and write, but
it would help me tune iopipe to cater to parsers, which I think is its
wheelhouse.
I plan to eventually finish the JSON parser for a releasable state, and
eventually tackle XML and a few other things.
-Steve
