Re: D and Async I/O
On Mon, 2020-05-18 at 11:56 +, Sebastiaan Koppe via Digitalmars-d-learn wrote: > On Thursday, 14 May 2020 at 09:36:33 UTC, Russel Winder wrote: > > Whilst C frameworks use callbacks and trampolines, high level > > languages seem to be basing things on futures – or things that > > are effectively isomorphic to futures. > > What I find most lacking is proper cancellation. Also, futures > are eager. Whilst GIO has an explicit cancellation stack parameter for all its async operations, I am not sure it is really necessary for the sort of applications I would write – I use null in all cases. Futures are neither eager nor lazy in and of themselves. I am not sure why this is an issue, futures are futures and promises are promises; they are what they are. > > Concurrency and parallelism will never be solved problems I > > suspect, but I think there is a fairly good consensus now on > > what is state of the art. > > I haven't found a language that ticks all the boxes. Kotlin comes > close. We have different needs. I am finding Python's asyncio/async/await and Rust's async/.await with futures perfectly reasonable for creating asynchronous (aka reactive in the hipster jargon) code. The Gio networking stuff is all very C. The Python PyGobject API to it doesn't seem to work, and the Rust API is still a work in progress (the client side works fine, the server-side needs work, and it all needs adding to gtk-rs. > > > I think there are a lot of lessons to be learned from all the > > > efforts in the programming community. > > > > > > We should: > > > > > > - get stackless coroutines > > > - get structured concurrency > > > - steal as many idea from the C++'s executors proposal > > > (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p0443r13.html) > > > > I am not convinced C++ has the best "state of the art" in this > > respect – after all it is just copying what JVM languages have > > had for ages, and Rust updated for modern native code languages. > > I am not sure you have read the proposal. Initially I brushed it > off, but upon closer inspection I realised there are some gems in > there. You are right. I will read it fully rather than just the abstract. Perhaps the C++ folk have learned lessons from the Kotlin, Python, and Rust stuff to have something better. I'm still not going to use C++, but it doesn't hurt to learn good lessons. The crucial need from my perspective is having channels, aka queues that create events on the event loop. gtk-rs has shown how to integrate futures and channels really quite well. -- Russel. === Dr Russel Winder t: +44 20 7585 2200 41 Buckmaster Roadm: +44 7770 465 077 London SW11 1EN, UK w: www.russel.org.uk signature.asc Description: This is a digitally signed message part
Re: D and Async I/O
On Thursday, 14 May 2020 at 09:36:33 UTC, Russel Winder wrote: Whilst C frameworks use callbacks and trampolines, high level languages seem to be basing things on futures – or things that are effectively isomorphic to futures. What I find most lacking is proper cancellation. Also, futures are eager. Concurrency and parallelism will never be solved problems I suspect, but I think there is a fairly good consensus now on what is state of the art. I haven't found a language that ticks all the boxes. Kotlin comes close. I think there are a lot of lessons to be learned from all the efforts in the programming community. We should: - get stackless coroutines - get structured concurrency - steal as many idea from the C++'s executors proposal (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p0443r13.html) I am not convinced C++ has the best "state of the art" in this respect – after all it is just copying what JVM languages have had for ages, and Rust updated for modern native code languages. I am not sure you have read the proposal. Initially I brushed it off, but upon closer inspection I realised there are some gems in there.
Re: D and Async I/O
On Tue, 2020-05-12 at 20:05 +0200, Jacob Carlborg via Digitalmars-d-learn wrote: > On 2020-05-12 11:23, Russel Winder wrote: > > > As far as I can tell D has no futures… > > Future and async in vibe.d [1]. Future in Mecca [2]. > > [1] https://vibed.org/api/vibe.core.concurrency/async > [2] > https://github.com/weka-io/mecca/blob/0593a35dd1a9978855d7db349fc1172f04cf8013/src/mecca/reactor/sync/future.d#L23 D needs something at the language level on which Vibe.d and Mecca build. This is a lesson from Rust, Kotlin, Python, etc. worth taking up. Unfortunately, I can't see D changing because it seems not enough people with interest in developing the language have the resource/interest in this. :-( -- Russel. === Dr Russel Winder t: +44 20 7585 2200 41 Buckmaster Roadm: +44 7770 465 077 London SW11 1EN, UK w: www.russel.org.uk signature.asc Description: This is a digitally signed message part
Re: D and Async I/O
On Tue, 2020-05-12 at 09:57 +, Sebastiaan Koppe via Digitalmars-d-learn wrote: […] > > Yeah it is a shame, but you see it in almost every language. > Probably means concurrency and io isn't a fully solved problem > yet. Whilst C frameworks use callbacks and trampolines, high level languages seem to be basing things on futures – or things that are effectively isomorphic to futures. Concurrency and parallelism will never be solved problems I suspect, but I think there is a fairly good consensus now on what is state of the art. D as a language is lagging, and this is sad. […] > > I think there are a lot of lessons to be learned from all the > efforts in the programming community. > > We should: > > - get stackless coroutines > - get structured concurrency > - steal as many idea from the C++'s executors proposal > (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p0443r13.html) I am not convinced C++ has the best "state of the art" in this respect – after all it is just copying what JVM languages have had for ages, and Rust updated for modern native code languages. But yes, if D doesn't get something in the async/await style then its future (!) is non-existent. :-( -- Russel. === Dr Russel Winder t: +44 20 7585 2200 41 Buckmaster Roadm: +44 7770 465 077 London SW11 1EN, UK w: www.russel.org.uk signature.asc Description: This is a digitally signed message part
Re: D and Async I/O
On 2020-05-12 11:23, Russel Winder wrote: As far as I can tell D has no futures… Future and async in vibe.d [1]. Future in Mecca [2]. [1] https://vibed.org/api/vibe.core.concurrency/async [2] https://github.com/weka-io/mecca/blob/0593a35dd1a9978855d7db349fc1172f04cf8013/src/mecca/reactor/sync/future.d#L23 -- /Jacob Carlborg
Re: D and Async I/O
On Tuesday, 12 May 2020 at 09:23:40 UTC, Russel Winder wrote: On Mon, 2020-05-11 at 19:34 +0200, Jacob Carlborg via Digitalmars-d-learn wrote: On 2020-05-11 16:44, Russel Winder wrote: > Crickey, a third option. This wil increase my dithering! ;-) Forth: Mecca [1] :) [1] https://github.com/weka-io/mecca Hummm… it seems everyone who needed async activity and particularly I/O in D has written their own. Mostly along with all their own data structures and algorithms library. Yeah it is a shame, but you see it in almost every language. Probably means concurrency and io isn't a fully solved problem yet. I keep trying to come back to D for GTK+ working, but in the end I keep going back to Python and Rust because D has no futures, and no added extras over GtkD auto translation of the GTK+ API to make it D-y in the way gtk-rs make GTK+ Rust-y. Sorry for the apparent gloom, I just felt the need to tell it how I feel. I think there are a lot of lessons to be learned from all the efforts in the programming community. We should: - get stackless coroutines - get structured concurrency - steal as many idea from the C++'s executors proposal (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p0443r13.html)
Re: D and Async I/O
On Mon, 2020-05-11 at 19:34 +0200, Jacob Carlborg via Digitalmars-d-learn wrote: > On 2020-05-11 16:44, Russel Winder wrote: > > > Crickey, a third option. This wil increase my dithering! ;-) > > Forth: Mecca [1] :) > > [1] https://github.com/weka-io/mecca Hummm… it seems everyone who needed async activity and particularly I/O in D has written their own. Mostly along with all their own data structures and algorithms library. The Rust experience is that there were also many attempts (cf. Tokio and Async_std) but that development and maintenance now seems focused on providing the minimal support for futures in the language (as an API to work with) and the crate futures to provide all the serious stuff, and that all the different event loops are converging on using this – Tokio and Async_std are moving to provide functionality over the std::futures and futures stuff as far as I can tell, indeed Async_std's name tells their story. It isn't pretty in many ways, but it works, and provides a one true Rust-y way of being asynchronous. gtk-rs is working to use the GTK+ async stuff (which is callback based) but provide it in a Tokio/Async_std kind of API based on std::futures and futures crate. This is a huge, huge plus over what D has. GtkD is missing all the added extras that gtk-rs is in the process of providing. As far as I can tell D has no futures… on which to base an equivalent system. I guess the async/.await language syntax will almost certainly never get into D even though it is the choice for Rust and Python – and indeed Kotlin but with a different syntax structure. But is there an alternative, a pure library based way. Clearly yes at the expense of some irritating verbosity that Rust, Python and Kotlin chose not to cope with, but to make language syntax changes instead. Of course this requires effort. Clearly, Rust, Python, and Kotlin have paid people to do all the futures stuff. Firther there is some effort to do this in gtk-rs and I am providing some input with this. If there was effort to add futures to D and extend GtkD in the way gtk-rs is being extended, it would be good for D. D is far, far better than Rust for writing GTK+ code, and could easily replace Vala. However, with the way gtk-rs is developing and GtkD is not, Rust will win out. Well at least people like me will use Rust and gtk-rs instead of D and GtkD because of the language and library evolution in the right direction. Sadly I think that whilst there may or may not be a flurry of activity on this thread, there will not be enough volunteers committed to do the work on futures in D and GtkD to make anything happen. I keep trying to come back to D for GTK+ working, but in the end I keep going back to Python and Rust because D has no futures, and no added extras over GtkD auto translation of the GTK+ API to make it D-y in the way gtk-rs make GTK+ Rust-y. Sorry for the apparent gloom, I just felt the need to tell it how I feel. -- Russel. === Dr Russel Winder t: +44 20 7585 2200 41 Buckmaster Roadm: +44 7770 465 077 London SW11 1EN, UK w: www.russel.org.uk signature.asc Description: This is a digitally signed message part
Re: D and Async I/O
On Monday, 11 May 2020 at 15:02:59 UTC, bauss wrote: On Monday, 11 May 2020 at 14:02:54 UTC, Russel Winder wrote: OK, so I need to create an asynchronous TCP server (not HTTP or HTTPS, this is a real server ;-) ). I think the normal response is "Use Vibe.d". However, recently I see Hunt is an alternative. Has anyone any way of choosing between the two? vibe.d is much more mature than Hunt, that would be my take on it. Also Hunt lacks documentation etc. I notice that Hunt uses it's own library eschewing all of Phobos. Is this an indicator that Phobos is not suitable for networking activity? std.socket is terrible, so yes that is an indicator. You can't even wrap something up fast in it either. Basically it's low-level while not being low-level at the same time. You have to handle __everything__ yourself pretty much. Have a look also to Martin std.io [1] and Steven iopipes [2], if you need something simple. [1] https://github.com/MartinNowak/io [2] https://code.dlang.org/packages/iopipe
Re: D and Async I/O
On Monday, 11 May 2020 at 21:15:28 UTC, Steven Schveighoffer
wrote:
On 5/11/20 3:46 PM, ikod wrote:
On Monday, 11 May 2020 at 17:34:41 UTC, Jacob Carlborg wrote:
On 2020-05-11 16:44, Russel Winder wrote:
Crickey, a third option. This wil increase my dithering! ;-)
Forth: Mecca [1] :)
[1] https://github.com/weka-io/mecca
And probably more. At least I also have my async library for
network IO.
It would be nice to have well defined interface for async io.
That will allow to choose and test different implementations.
Part of the problem is the different APIs that async libraries
use. Some use callbacks, some use fibers, maybe some
async/await forms.
callbacks are basic building blocks, which should be supported
anyway.
std.io aims to provide a common interface for async and sync
i/o, where you choose the driver at the beginning of your
program. If written properly, it could be a nice way to test
My library also can work in this modes. There are low-level
"sockets" which can be used only with callbacks, and also
high-level sockets which can be used in "sync" mode in fibers or
even without fibers as a replacement for std.socket's. Here is
simple example where callback-based Timer used to build "sync"
sleep:
void Sleep(Duration d) {
if ( d <= 0.seconds) {
return;
}
auto f = Fiber.getThis();
if (f is null)
{
// called not from fiber/task, there is no concurrency,
// use plain old sleep
Thread.sleep(d);
return;
}
// otherwise we yield and return to current fiber
// later, when timer expires (using event loop)
auto callback = delegate void (AppEvent e)
{
assert(e == TimeoutExpired);
f.call();
};
auto t = new Timer(d, callback);
getDefaultLoop().startTimer(t);
Fiber.yield();
}
code with various drivers without having to change code. It
would require a Fiber-based approach, however.
I have not added an async driver (yet), but it's in my somewhat
immediate plans to do so, as I want to start using this more
(along with iopipe). Contributions would be welcome.
initially very few interfaces required - like start/stop for
Timer, start/stop for SignalHandling, start/stop for file/socket
poll.
https://github.com/MartinNowak/io
-Steve
Re: D and Async I/O
On 5/11/20 3:46 PM, ikod wrote: On Monday, 11 May 2020 at 17:34:41 UTC, Jacob Carlborg wrote: On 2020-05-11 16:44, Russel Winder wrote: Crickey, a third option. This wil increase my dithering! ;-) Forth: Mecca [1] :) [1] https://github.com/weka-io/mecca And probably more. At least I also have my async library for network IO. It would be nice to have well defined interface for async io. That will allow to choose and test different implementations. Part of the problem is the different APIs that async libraries use. Some use callbacks, some use fibers, maybe some async/await forms. std.io aims to provide a common interface for async and sync i/o, where you choose the driver at the beginning of your program. If written properly, it could be a nice way to test code with various drivers without having to change code. It would require a Fiber-based approach, however. I have not added an async driver (yet), but it's in my somewhat immediate plans to do so, as I want to start using this more (along with iopipe). Contributions would be welcome. https://github.com/MartinNowak/io -Steve
Re: D and Async I/O
On Monday, 11 May 2020 at 17:34:41 UTC, Jacob Carlborg wrote: On 2020-05-11 16:44, Russel Winder wrote: Crickey, a third option. This wil increase my dithering! ;-) Forth: Mecca [1] :) [1] https://github.com/weka-io/mecca And probably more. At least I also have my async library for network IO. It would be nice to have well defined interface for async io. That will allow to choose and test different implementations.
Re: D and Async I/O
On 2020-05-11 16:44, Russel Winder wrote: Crickey, a third option. This wil increase my dithering! ;-) Forth: Mecca [1] :) [1] https://github.com/weka-io/mecca -- /Jacob Carlborg
Re: D and Async I/O
On Monday, 11 May 2020 at 14:02:54 UTC, Russel Winder wrote: OK, so I need to create an asynchronous TCP server (not HTTP or HTTPS, this is a real server ;-) ). I think the normal response is "Use Vibe.d". However, recently I see Hunt is an alternative. Has anyone any way of choosing between the two? vibe.d is much more mature than Hunt, that would be my take on it. Also Hunt lacks documentation etc. I notice that Hunt uses it's own library eschewing all of Phobos. Is this an indicator that Phobos is not suitable for networking activity? std.socket is terrible, so yes that is an indicator. You can't even wrap something up fast in it either. Basically it's low-level while not being low-level at the same time. You have to handle __everything__ yourself pretty much.
Re: D and Async I/O
On Mon, 2020-05-11 at 16:36 +0200, Daniel Kozak via Digitalmars-d-learn wrote: > On Mon, May 11, 2020 at 4:03 PM Russel Winder via Digitalmars-d-learn > wrote: > > ... > > I notice that Hunt uses it's own library eschewing all of Phobos. Is this > > an > > indicator that Phobos is not suitable for networking activity? > Vibe-d do that too, But https://code.dlang.org/packages/async use > phobos socket and works well Crickey, a third option. This wil increase my dithering! ;-) -- Russel. === Dr Russel Winder t: +44 20 7585 2200 41 Buckmaster Roadm: +44 7770 465 077 London SW11 1EN, UK w: www.russel.org.uk signature.asc Description: This is a digitally signed message part
Re: D and Async I/O
On Mon, 2020-05-11 at 15:02 +0100, Russel Winder wrote: > OK, so I need to create an asynchronous TCP server (not HTTP or HTTPS, this > is > a real server ;-) ). > > I think the normal response is "Use Vibe.d". However, recently I see Hunt is > an alternative. Has anyone any way of choosing between the two? > > I notice that Hunt uses it's own library eschewing all of Phobos. Is this an > indicator that Phobos is not suitable for networking activity? Of course the really obvious solution would be to use the GTK+ event loop and GtkD API binding even though the server has no UI since it is a server that is part of the integration tests for a GtkD realised GUI desktop application. -- Russel. === Dr Russel Winder t: +44 20 7585 2200 41 Buckmaster Roadm: +44 7770 465 077 London SW11 1EN, UK w: www.russel.org.uk signature.asc Description: This is a digitally signed message part
Re: D and Async I/O
On Mon, May 11, 2020 at 4:03 PM Russel Winder via Digitalmars-d-learn wrote: > > ... > I notice that Hunt uses it's own library eschewing all of Phobos. Is this an > indicator that Phobos is not suitable for networking activity? Vibe-d do that too, But https://code.dlang.org/packages/async use phobos socket and works well
