On Tuesday, 12 June 2018 at 13:51:56 UTC, Steven Schveighoffer
wrote:
On 6/12/18 1:51 AM, DigitalDesigns wrote:
Could you explain some benefits specific to this
implementation and a bit of the functional aspects for a
proper overview of it's capabilities and why I should chose
this method over others?
The things that I think make this approach better are:
1. Direct buffer access
Direct buffer access I have found is one of those ideas that
doesn't seem like it's that impressive until you start using
it. Many times, buffering is used in a generic library for
optimization (basically amortizing reads) and is an
implementation detail hidden from your view. Think of how FILE
* keeps a large buffer of data inside itself, but only gives
you access to one character at a time.
This forces you to create your *own* buffering scheme on top of
that. What a waste! Iopipe allows you to use buffering for your
purposes on top of the benefits of amortization. It's my belief
that this is why iopipe's byline feature is 2x faster than
Phobos'.
2. Using templates to their fullest
Iopipes are all templated on the buffer or iopipe underneath
it. This makes tings easily swappable. It's really cool to be
able to take your JSON or XML parser, and hook it onto an
in-memory string in one line, and then hook it onto a socket,
and everything is optimized for that situation. It takes the
fun and flexibility of range programming and brings it to i/o.
This is why iopipe's byline handles all forms of UTF, compared
to Phobos which only handles UTF8.
For example, I handle all forms of UTF with iopipe, with a
decent set of utilities. Here is a complete program using
iopipe that converts any type of UTF into another type,
optimized for the specific situation:
https://github.com/schveiguy/iopipe/blob/master/examples/convert/convert.d
3. Compiler optimization for everything
All parts of iopipe, except for the low-level reads and writes
(which ironically are not really part of iopipe) are visible to
the compiler for inlining and optimization. I'm leveraging the
power of the decades of optimization experience that the
compiler can provide. This makes it easy to write code that
performs well.
An anecdote: For my talk on iopipe in 2017
(http://dconf.org/2017/talks/schveighoffer.html) I wanted to
have a live demo showing the performance power. I literally was
still working on it 2 or 3 days before, while at dconf. I had
already written a JSON parser, which was part of my
presentation, but when I was showing it to another D user
(Daniel Murphy), I couldn't really answer the question "how
does it perform?". So he gave me a challenge -- do pretty
printing on a JSON file. Simple enough, with the parser I had
already written, took me about 1 hour to write it. It performed
poorly compared to what we would have expected, but tweaking a
few things (almost all were due to using some algorithms
incorrectly), I got it to go faster than RapidJson in certain
use cases, and reasonably close in others. And I did nothing in
terms of lookup tables or using any special instructions. All
in all, it was probably 2 hours of work, and the code is
beautiful IMO!
https://github.com/schveiguy/jsoniopipe/blob/master/examples/formatjson/formatjson.d
I think anyone who is doing parsing should have a look at
iopipe, it not only may make your code much simpler and easier
to read and write, but it would help me tune iopipe to cater to
parsers, which I think is its wheelhouse.
I plan to eventually finish the JSON parser for a releasable
state, and eventually tackle XML and a few other things.
-Steve
Thanks!
