Re: iopipe v0.1.0 - now with Windows support!
On Tuesday, 19 June 2018 at 13:04:24 UTC, Steven Schveighoffer wrote: On 6/19/18 7:18 AM, Jacob Carlborg wrote: On 2018-06-11 16:45, Steven Schveighoffer wrote: I just pushed v0.1.1 -- I realized that I never *actually* compiled on windows, and there were a couple things that didn't work. Note: the examples still don't work as they rely on openDev, which is only available on Posix systems now. I need to figure out a good way to open stdin/stdout in a cross platform way with std.io. You should setup AppVeyor [1] to make it works on Windows (when it works). [1] https://www.appveyor.com I just set up travis to do the Linux/mac testing. I need to add appveyor as well, but haven't gotten to it. I'm a complete CI noob, so I'm learning slowly :) -Steve Just copy/paste one of the existing configuration scripts and you should be good to go, e.g. https://github.com/dlang-community/DCD/blob/master/appveyor.yml And there's also: https://github.com/Abscissa/AppVeyor-D
Re: iopipe v0.1.0 - now with Windows support!
On 2018-06-19 15:04, Steven Schveighoffer wrote: I just set up travis to do the Linux/mac testing. I need to add appveyor as well, but haven't gotten to it. I'm a complete CI noob, so I'm learning slowly :) To save you some trouble, AppVeyor supports both a YAML, like Travis, and a web UI to configure the CI system. If you're not including some parts of the YAML file, like "build_script", it will use the default, which is preforming some Visual Studio specific task. You also need to download the D compiler manually since AppVeyor doesn't have built-in support for D the same way as Travis. -- /Jacob Carlborg
Re: iopipe v0.1.0 - now with Windows support!
On 6/19/18 7:18 AM, Jacob Carlborg wrote: On 2018-06-11 16:45, Steven Schveighoffer wrote: I just pushed v0.1.1 -- I realized that I never *actually* compiled on windows, and there were a couple things that didn't work. Note: the examples still don't work as they rely on openDev, which is only available on Posix systems now. I need to figure out a good way to open stdin/stdout in a cross platform way with std.io. You should setup AppVeyor [1] to make it works on Windows (when it works). [1] https://www.appveyor.com I just set up travis to do the Linux/mac testing. I need to add appveyor as well, but haven't gotten to it. I'm a complete CI noob, so I'm learning slowly :) -Steve
Re: iopipe v0.1.0 - now with Windows support!
On 2018-06-11 16:45, Steven Schveighoffer wrote: I just pushed v0.1.1 -- I realized that I never *actually* compiled on windows, and there were a couple things that didn't work. Note: the examples still don't work as they rely on openDev, which is only available on Posix systems now. I need to figure out a good way to open stdin/stdout in a cross platform way with std.io. You should setup AppVeyor [1] to make it works on Windows (when it works). [1] https://www.appveyor.com -- /Jacob Carlborg
Re: iopipe v0.1.0 - now with Windows support!
On Sunday, 17 June 2018 at 04:52:07 UTC, Martin Nowak wrote: On 06/10/2018 10:10 PM, Steven Schveighoffer wrote: Note that the new io library also supports sockets, which IODev did not have support for, AND has a pluggable driver system, so you could potentially use fiber-based async io without rebuilding. It just makes a lot of sense for D to have a standard low-level io library that everything can use without having to kludge together multiple types of io libraries. Note that the WIP std.io library is fully @nogc @safe, so it's a bit edgy on using latest features. Soon want to move to use DIP10008 instead of preallocated exceptions. This is very encouraging. I’d like to see it working well with Photon (though my time is very limited atm). Any thoughts on what set of syscalls I need to support? Maybe I could just provide my own “native” driver that fits your concept of I/O driver in io library. With that and @nogc in the Driver interface¹ it's still to be seen whether we can adapt this well with vibe.d or need to adjust the low-level design. -Martin ¹: https://martinnowak.github.io/io/std/io/driver/Driver.html
Re: iopipe v0.1.0 - now with Windows support!
On 06/10/2018 10:10 PM, Steven Schveighoffer wrote: > Note that the new io library also supports sockets, which IODev did not > have support for, AND has a pluggable driver system, so you could > potentially use fiber-based async io without rebuilding. It just makes a > lot of sense for D to have a standard low-level io library that > everything can use without having to kludge together multiple types of > io libraries. Note that the WIP std.io library is fully @nogc @safe, so it's a bit edgy on using latest features. Soon want to move to use DIP10008 instead of preallocated exceptions. With that and @nogc in the Driver interface¹ it's still to be seen whether we can adapt this well with vibe.d or need to adjust the low-level design. -Martin ¹: https://martinnowak.github.io/io/std/io/driver/Driver.html
Re: iopipe v0.1.0 - now with Windows support!
On 6/13/18 12:03 PM, Steven Schveighoffer wrote: I'm going to push this (I'll do some tests for the other widths to make sure it works for all UTF), but if you have any more things you want to work at CTFE, submit some issues on the github project. v0.1.2 released -Steve
Re: iopipe v0.1.0 - now with Windows support!
On 6/13/18 8:35 AM, bauss wrote:
Does iopipe work with CTFE?
It may work in some cases. Some of the things it does are not conducive
to CTFE working well -- like using a buffer. But generally at compile
time, you don't want to use a buffer.
But I would expect, for instance, using algorithms and iopipes on a
string would probably work.
hm... just thought I'd try it:
import std.range : walkLength;
static
assert("hello\nworld\nthis\nis\na\ntest".byLineRange.walkLength == 6);
It didn't work at first, as I'm using memchr for searching for the
newlines, but with a nice little if(__ctfe) override, it works just great!
I'm going to push this (I'll do some tests for the other widths to make
sure it works for all UTF), but if you have any more things you want to
work at CTFE, submit some issues on the github project.
I don't expect a lot of the array casting stuff to work, but maybe there
are ways. I actually never thought much about making it work at
compile-time. I suppose though, it would be cool to import a file at
compile time, and generate the JSON or XML DOM at compile time too :)
I think zip and unzip are never going to work since the underlying C
calls are not CTFE-able.
-Steve
Re: iopipe v0.1.0 - now with Windows support!
On Sunday, 10 June 2018 at 20:10:31 UTC, Steven Schveighoffer wrote: -Steve Does iopipe work with CTFE?
Re: iopipe v0.1.0 - now with Windows support!
On Tuesday, 12 June 2018 at 14:00:32 UTC, Steven Schveighoffer wrote: File.byLine is fast, but only because of the underlying non-range tricks it uses to achieve performance. And iopipe still is 2x faster. I wish iopipe was around a little bit earlier so I could use it in my small project. It doesn't use IO much, just reads large file (60GB+) and computes some hashes. Now I'd like to rewrite it using iopipe just to compare the performance.
Re: iopipe v0.1.0 - now with Windows support!
On Tuesday, 12 June 2018 at 13:51:56 UTC, Steven Schveighoffer wrote: On 6/12/18 1:51 AM, DigitalDesigns wrote: Could you explain some benefits specific to this implementation and a bit of the functional aspects for a proper overview of it's capabilities and why I should chose this method over others? The things that I think make this approach better are: 1. Direct buffer access Direct buffer access I have found is one of those ideas that doesn't seem like it's that impressive until you start using it. Many times, buffering is used in a generic library for optimization (basically amortizing reads) and is an implementation detail hidden from your view. Think of how FILE * keeps a large buffer of data inside itself, but only gives you access to one character at a time. This forces you to create your *own* buffering scheme on top of that. What a waste! Iopipe allows you to use buffering for your purposes on top of the benefits of amortization. It's my belief that this is why iopipe's byline feature is 2x faster than Phobos'. 2. Using templates to their fullest Iopipes are all templated on the buffer or iopipe underneath it. This makes tings easily swappable. It's really cool to be able to take your JSON or XML parser, and hook it onto an in-memory string in one line, and then hook it onto a socket, and everything is optimized for that situation. It takes the fun and flexibility of range programming and brings it to i/o. This is why iopipe's byline handles all forms of UTF, compared to Phobos which only handles UTF8. For example, I handle all forms of UTF with iopipe, with a decent set of utilities. Here is a complete program using iopipe that converts any type of UTF into another type, optimized for the specific situation: https://github.com/schveiguy/iopipe/blob/master/examples/convert/convert.d 3. Compiler optimization for everything All parts of iopipe, except for the low-level reads and writes (which ironically are not really part of iopipe) are visible to the compiler for inlining and optimization. I'm leveraging the power of the decades of optimization experience that the compiler can provide. This makes it easy to write code that performs well. An anecdote: For my talk on iopipe in 2017 (http://dconf.org/2017/talks/schveighoffer.html) I wanted to have a live demo showing the performance power. I literally was still working on it 2 or 3 days before, while at dconf. I had already written a JSON parser, which was part of my presentation, but when I was showing it to another D user (Daniel Murphy), I couldn't really answer the question "how does it perform?". So he gave me a challenge -- do pretty printing on a JSON file. Simple enough, with the parser I had already written, took me about 1 hour to write it. It performed poorly compared to what we would have expected, but tweaking a few things (almost all were due to using some algorithms incorrectly), I got it to go faster than RapidJson in certain use cases, and reasonably close in others. And I did nothing in terms of lookup tables or using any special instructions. All in all, it was probably 2 hours of work, and the code is beautiful IMO! https://github.com/schveiguy/jsoniopipe/blob/master/examples/formatjson/formatjson.d I think anyone who is doing parsing should have a look at iopipe, it not only may make your code much simpler and easier to read and write, but it would help me tune iopipe to cater to parsers, which I think is its wheelhouse. I plan to eventually finish the JSON parser for a releasable state, and eventually tackle XML and a few other things. -Steve Thanks!
Re: iopipe v0.1.0 - now with Windows support!
On 6/12/18 10:19 AM, Jesse Phillips wrote: I plan to eventually finish the JSON parser for a releasable state, and eventually tackle XML and a few other things. You should definitely tackle xml by branching dxml. I'm really liking the api. Of course that is my plan! I would never want to have to build an xml parser from scratch :) -Steve
Re: iopipe v0.1.0 - now with Windows support!
I plan to eventually finish the JSON parser for a releasable state, and eventually tackle XML and a few other things. -Steve You should definitely tackle xml by branching dxml. I'm really liking the api.
Re: iopipe v0.1.0 - now with Windows support!
On 6/12/18 3:08 AM, Anton Fediushin wrote: On Sunday, 10 June 2018 at 20:10:31 UTC, Steven Schveighoffer wrote: iopipe version 0.1.0 has been released. iopipe is a high-performance pipe processing system that makes it easy to string together pipelines to process data with as little buffer copying as possible. I saw iopipe a while back, but never looked at it closely. Now I did and... it implements its own kind of pattern and not ranges? Correct, although it's very similar to ranges. I guess it is done for the sake of performance, but how easy it is to use iopipe with standard range-based methods from std.algorithm for example? Very simple. You just have to define what is an "element" of a range that is a sliding window of data. For example: https://github.com/schveiguy/iopipe/blob/master/source/iopipe/textpipe.d#L542 Note that asInputRange simply treats the current window as "front", and "popFront" discards that window and loads the next. It's a crude but effective tool to convert any iopipe into a range. The reason iopipe is not based on ranges exactly (the window must be a random-access range), is because ranges don't handle i/o very performantly. E.g. I would never use lockingTextReader to process text data, it would be slow as hell, and too limiting. File.byLine is fast, but only because of the underlying non-range tricks it uses to achieve performance. And iopipe still is 2x faster. As long as it's easy to use with the rest of the phobos - I'd like to see it in the standard library. It should be easy to use on its own, and with algorithms from phobos. I've done so in some of the toy parsers I've written. I think the one sticking point (which I'm not sure how to reconcile, but we can probably hash it out) is that for iopipes, char and wchar arrays are arrays, not auto-decoding ranges. -Steve
Re: iopipe v0.1.0 - now with Windows support!
On 6/12/18 1:51 AM, DigitalDesigns wrote: Could you explain some benefits specific to this implementation and a bit of the functional aspects for a proper overview of it's capabilities and why I should chose this method over others? The things that I think make this approach better are: 1. Direct buffer access Direct buffer access I have found is one of those ideas that doesn't seem like it's that impressive until you start using it. Many times, buffering is used in a generic library for optimization (basically amortizing reads) and is an implementation detail hidden from your view. Think of how FILE * keeps a large buffer of data inside itself, but only gives you access to one character at a time. This forces you to create your *own* buffering scheme on top of that. What a waste! Iopipe allows you to use buffering for your purposes on top of the benefits of amortization. It's my belief that this is why iopipe's byline feature is 2x faster than Phobos'. 2. Using templates to their fullest Iopipes are all templated on the buffer or iopipe underneath it. This makes tings easily swappable. It's really cool to be able to take your JSON or XML parser, and hook it onto an in-memory string in one line, and then hook it onto a socket, and everything is optimized for that situation. It takes the fun and flexibility of range programming and brings it to i/o. This is why iopipe's byline handles all forms of UTF, compared to Phobos which only handles UTF8. For example, I handle all forms of UTF with iopipe, with a decent set of utilities. Here is a complete program using iopipe that converts any type of UTF into another type, optimized for the specific situation: https://github.com/schveiguy/iopipe/blob/master/examples/convert/convert.d 3. Compiler optimization for everything All parts of iopipe, except for the low-level reads and writes (which ironically are not really part of iopipe) are visible to the compiler for inlining and optimization. I'm leveraging the power of the decades of optimization experience that the compiler can provide. This makes it easy to write code that performs well. An anecdote: For my talk on iopipe in 2017 (http://dconf.org/2017/talks/schveighoffer.html) I wanted to have a live demo showing the performance power. I literally was still working on it 2 or 3 days before, while at dconf. I had already written a JSON parser, which was part of my presentation, but when I was showing it to another D user (Daniel Murphy), I couldn't really answer the question "how does it perform?". So he gave me a challenge -- do pretty printing on a JSON file. Simple enough, with the parser I had already written, took me about 1 hour to write it. It performed poorly compared to what we would have expected, but tweaking a few things (almost all were due to using some algorithms incorrectly), I got it to go faster than RapidJson in certain use cases, and reasonably close in others. And I did nothing in terms of lookup tables or using any special instructions. All in all, it was probably 2 hours of work, and the code is beautiful IMO! https://github.com/schveiguy/jsoniopipe/blob/master/examples/formatjson/formatjson.d I think anyone who is doing parsing should have a look at iopipe, it not only may make your code much simpler and easier to read and write, but it would help me tune iopipe to cater to parsers, which I think is its wheelhouse. I plan to eventually finish the JSON parser for a releasable state, and eventually tackle XML and a few other things. -Steve
Re: iopipe v0.1.0 - now with Windows support!
On Sunday, 10 June 2018 at 20:10:31 UTC, Steven Schveighoffer wrote: iopipe version 0.1.0 has been released. iopipe is a high-performance pipe processing system that makes it easy to string together pipelines to process data with as little buffer copying as possible. I saw iopipe a while back, but never looked at it closely. Now I did and... it implements its own kind of pattern and not ranges? I guess it is done for the sake of performance, but how easy it is to use iopipe with standard range-based methods from std.algorithm for example? As long as it's easy to use with the rest of the phobos - I'd like to see it in the standard library. This version ONLY builds on 2.080.1 or later (the bug fix that I submitted at dconf has been merged in that version, and so iopipe will now build against Martin Nowak's io library). In fact, iopipe development was kind of stalled due to this bug, so I'm super-happy to see it fixed and released! These are great news, keep it up!
Re: iopipe v0.1.0 - now with Windows support!
On Sunday, 10 June 2018 at 20:10:31 UTC, Steven Schveighoffer wrote: iopipe version 0.1.0 has been released. iopipe is a high-performance pipe processing system that makes it easy to string together pipelines to process data with as little buffer copying as possible. Nothing has really been changed, but it now has Windows i/o support. I will note at this time, however, that ring buffers are not yet supported on Windows. This version deprecates the IODev type that I had included, in favor of a new io library that shows extreme promise. This version ONLY builds on 2.080.1 or later (the bug fix that I submitted at dconf has been merged in that version, and so iopipe will now build against Martin Nowak's io library). In fact, iopipe development was kind of stalled due to this bug, so I'm super-happy to see it fixed and released! Note that the new io library also supports sockets, which IODev did not have support for, AND has a pluggable driver system, so you could potentially use fiber-based async io without rebuilding. It just makes a lot of sense for D to have a standard low-level io library that everything can use without having to kludge together multiple types of io libraries. Near future plans: 1. Utilize a CI to make sure it continues to work on all platforms. 2. Add RingBuffer support on Windows 3. Add more driver support for std.io. 4. Continue development of JSON library that depends on iopipe (not yet on code.dlang.org). git - https://github.com/schveiguy/iopipe dub - https://code.dlang.org/packages/iopipe docs - http://schveiguy.github.io/iopipe/ -Steve Could you explain some benefits specific to this implementation and a bit of the functional aspects for a proper overview of it's capabilities and why I should chose this method over others?
Re: iopipe v0.1.0 - now with Windows support!
On 6/10/18 4:10 PM, Steven Schveighoffer wrote: Nothing has really been changed, but it now has Windows i/o support. I will note at this time, however, that ring buffers are not yet supported on Windows. I just pushed v0.1.1 -- I realized that I never *actually* compiled on windows, and there were a couple things that didn't work. Note: the examples still don't work as they rely on openDev, which is only available on Posix systems now. I need to figure out a good way to open stdin/stdout in a cross platform way with std.io. -Steve
Re: iopipe v0.1.0 - now with Windows support!
On 6/11/18 6:14 AM, Dejan Lekic wrote: On Sunday, 10 June 2018 at 20:10:31 UTC, Steven Schveighoffer wrote: iopipe version 0.1.0 has been released. iopipe is a high-performance pipe processing system that makes it easy to string together pipelines to process data with as little buffer copying as possible. All I can say (again, like I repeated on IRC many times) is that iopipe should be in the current form, or another, in Phobos. I just love it! Thanks for the kind words! If Martin's std.io makes it in to Phobos, there is a realistic chance iopipe could go as well. Just curious, do you have any projects based on iopipe? -Steve
Re: iopipe v0.1.0 - now with Windows support!
On Sunday, 10 June 2018 at 20:10:31 UTC, Steven Schveighoffer wrote: iopipe version 0.1.0 has been released. iopipe is a high-performance pipe processing system that makes it easy to string together pipelines to process data with as little buffer copying as possible. All I can say (again, like I repeated on IRC many times) is that iopipe should be in the current form, or another, in Phobos. I just love it!
iopipe v0.1.0 - now with Windows support!
iopipe version 0.1.0 has been released. iopipe is a high-performance pipe processing system that makes it easy to string together pipelines to process data with as little buffer copying as possible. Nothing has really been changed, but it now has Windows i/o support. I will note at this time, however, that ring buffers are not yet supported on Windows. This version deprecates the IODev type that I had included, in favor of a new io library that shows extreme promise. This version ONLY builds on 2.080.1 or later (the bug fix that I submitted at dconf has been merged in that version, and so iopipe will now build against Martin Nowak's io library). In fact, iopipe development was kind of stalled due to this bug, so I'm super-happy to see it fixed and released! Note that the new io library also supports sockets, which IODev did not have support for, AND has a pluggable driver system, so you could potentially use fiber-based async io without rebuilding. It just makes a lot of sense for D to have a standard low-level io library that everything can use without having to kludge together multiple types of io libraries. Near future plans: 1. Utilize a CI to make sure it continues to work on all platforms. 2. Add RingBuffer support on Windows 3. Add more driver support for std.io. 4. Continue development of JSON library that depends on iopipe (not yet on code.dlang.org). git - https://github.com/schveiguy/iopipe dub - https://code.dlang.org/packages/iopipe docs - http://schveiguy.github.io/iopipe/ -Steve
