On 6/12/18 1:51 AM, DigitalDesigns wrote:

Could you explain some benefits specific to this implementation and a bit of the functional aspects for a proper overview of it's capabilities and why I should chose this method over others?

The things that I think make this approach better are:

1. Direct buffer access

Direct buffer access I have found is one of those ideas that doesn't seem like it's that impressive until you start using it. Many times, buffering is used in a generic library for optimization (basically amortizing reads) and is an implementation detail hidden from your view. Think of how FILE * keeps a large buffer of data inside itself, but only gives you access to one character at a time.

This forces you to create your *own* buffering scheme on top of that. What a waste! Iopipe allows you to use buffering for your purposes on top of the benefits of amortization. It's my belief that this is why iopipe's byline feature is 2x faster than Phobos'.

2. Using templates to their fullest

Iopipes are all templated on the buffer or iopipe underneath it. This makes tings easily swappable. It's really cool to be able to take your JSON or XML parser, and hook it onto an in-memory string in one line, and then hook it onto a socket, and everything is optimized for that situation. It takes the fun and flexibility of range programming and brings it to i/o.

This is why iopipe's byline handles all forms of UTF, compared to Phobos which only handles UTF8.

For example, I handle all forms of UTF with iopipe, with a decent set of utilities. Here is a complete program using iopipe that converts any type of UTF into another type, optimized for the specific situation:


3. Compiler optimization for everything

All parts of iopipe, except for the low-level reads and writes (which ironically are not really part of iopipe) are visible to the compiler for inlining and optimization. I'm leveraging the power of the decades of optimization experience that the compiler can provide. This makes it easy to write code that performs well.

An anecdote: For my talk on iopipe in 2017 (http://dconf.org/2017/talks/schveighoffer.html) I wanted to have a live demo showing the performance power. I literally was still working on it 2 or 3 days before, while at dconf. I had already written a JSON parser, which was part of my presentation, but when I was showing it to another D user (Daniel Murphy), I couldn't really answer the question "how does it perform?". So he gave me a challenge -- do pretty printing on a JSON file. Simple enough, with the parser I had already written, took me about 1 hour to write it. It performed poorly compared to what we would have expected, but tweaking a few things (almost all were due to using some algorithms incorrectly), I got it to go faster than RapidJson in certain use cases, and reasonably close in others. And I did nothing in terms of lookup tables or using any special instructions. All in all, it was probably 2 hours of work, and the code is beautiful IMO! https://github.com/schveiguy/jsoniopipe/blob/master/examples/formatjson/formatjson.d

I think anyone who is doing parsing should have a look at iopipe, it not only may make your code much simpler and easier to read and write, but it would help me tune iopipe to cater to parsers, which I think is its wheelhouse.

I plan to eventually finish the JSON parser for a releasable state, and eventually tackle XML and a few other things.


Reply via email to