Re: [rust-dev] Update on I/O progress

2013-04-30 Thread james
On 29/04/2013 20:40, Graydon Hoare wrote: On 13-04-28 12:05 PM, james wrote: Well, the issue is whether the program logic that will deal with data calls (say) read and iterates over the response, or whether something gets data and invokes a callback, in which case the code that deals with the

Re: [rust-dev] Update on I/O progress

2013-04-30 Thread Graydon Hoare
On 30/04/2013 12:17 PM, james wrote: This is what I call blocking synchronous read, a 'pull' model. Its hardly different from a threaded rogram with a userspace switch and sync to async call translation, whether N:1 or N:M. Granted that the compiler support for task switch points and segmented

Re: [rust-dev] Update on I/O progress

2013-04-30 Thread Brian Anderson
On 04/26/2013 04:40 PM, Vadim wrote: I share your concern. It's likely that Rust tasks will never scale as well as Erlang and I suspect that we will ultimately want an async API that does not impose the expense of 1 task per I/O stream. I hope though that the current approach

Re: [rust-dev] Update on I/O progress

2013-04-30 Thread Brian Anderson
On 04/27/2013 03:07 AM, Jordi Boggiano wrote: Take my input with a grain of salt since I'm so new to the language, but here goes nothing: I am particularly interested in making sure that common operations are convenient and don't require a lot of boilerplate. To that end I've been collecting

Re: [rust-dev] Update on I/O progress

2013-04-30 Thread Vadim
I see, thank you for clarification. Can't help but note, though, that the dynamic loading case could be worked around by having a wrapper similar to the one Rust generates around calls to foreign functions, that would ensure bigger stack segment for the dynamic linker to use (only the first time

Re: [rust-dev] Update on I/O progress

2013-04-29 Thread Graydon Hoare
On 13-04-28 12:05 PM, james wrote: Well, the issue is whether the program logic that will deal with data calls (say) read and iterates over the response, or whether something gets data and invokes a callback, in which case the code that deals with the data only has a thread (or stack if you

Re: [rust-dev] Update on I/O progress

2013-04-29 Thread Gábor Lehel
On Fri, Apr 26, 2013 at 11:11 PM, Brian Anderson bander...@mozilla.comwrote: On 04/26/2013 09:04 AM, Patrick Walton wrote: On 4/26/13 6:53 AM, Gábor Lehel wrote: Probably a crazy idea, but would it be possible to do something where if the parent task launches a child task, and then does

Re: [rust-dev] Update on I/O progress

2013-04-28 Thread james
On 26/04/2013 17:06, Patrick Walton wrote: On 4/25/13 10:42 PM, james wrote: If not, I wonder if there should be more effort on the AIO layer first, and then consider a synchronous shim on top. That's exactly what's being done. Patrick OK. It wasn't clear whether the AIO layer was

Re: [rust-dev] Update on I/O progress

2013-04-28 Thread james
It seems to me that, by default, what we'd want is a sync api which under the hood, creates one task per drive accessed, or maybe max(drives_in_use, number_of_cores). I think the OS will make them async with the drive hardware anyway, though, so number_of_cores doesn't matter. I don't think

Re: [rust-dev] Update on I/O progress

2013-04-27 Thread Jordi Boggiano
Take my input with a grain of salt since I'm so new to the language, but here goes nothing: I am particularly interested in making sure that common operations are convenient and don't require a lot of boilerplate. To that end I've been collecting [aspirational examples] of succinctly written

Re: [rust-dev] Update on I/O progress

2013-04-27 Thread David Bruant
Le 26/04/2013 23:23, Brian Anderson a écrit : On 04/25/2013 10:42 PM, james wrote: On 24/04/2013 23:38, Brian Anderson wrote: What I am focused on now is the design of a synchronous API for sending and receiving streams of bytes, along with an implementation built on an internal

Re: [rust-dev] Update on I/O progress

2013-04-27 Thread Lee Braiden
On 27/04/13 12:06, David Bruant wrote: Le 26/04/2013 23:23, Brian Anderson a écrit : I do think 50,000 I/O-performing tasks in a 32-bit process is within our reach. Last year I demonstrated 500,000 simultaneous Rust tasks (that did nothing but wait) in a 32-bit process, but there have been

Re: [rust-dev] Update on I/O progress

2013-04-27 Thread Daniel Micay
On Sat, Apr 27, 2013 at 12:10 PM, Lee Braiden leebr...@gmail.com wrote: On 27/04/13 12:06, David Bruant wrote: Le 26/04/2013 23:23, Brian Anderson a écrit : I do think 50,000 I/O-performing tasks in a 32-bit process is within our reach. Last year I demonstrated 500,000 simultaneous Rust

Re: [rust-dev] Update on I/O progress

2013-04-27 Thread Patrick Walton
On 4/27/13 4:06 AM, David Bruant wrote: I believe too this is doomed to happen. Firefox is under a huge struggle to remove as much as sync IO code as possible as part of the Snappy effort [1]. This is a completely misleading comparison. Firefox's sync I/O *blocks the entire main thread of the

Re: [rust-dev] Update on I/O progress

2013-04-27 Thread Lee Braiden
Jordi Boggiano Wrote: Feels weird to have str implement something specific to another module, but if there is no other way it sounds more user friendly than a gazillion open_* methods. A few more comments on other things: - Streams are great, and the ability to register new stream protocol

Re: [rust-dev] Update on I/O progress

2013-04-27 Thread Mikhail Zabaluev
Hi Lee, 2013/4/27 Lee Braiden leebr...@gmail.com: But we're not talking about creating a new task for every io operation as the io module's usual under-the-hood strategy, surely? It seems to me that, by default, what we'd want is a sync api which under the hood, creates one task per drive

Re: [rust-dev] Update on I/O progress

2013-04-27 Thread David Bruant
Le 27/04/2013 19:30, Patrick Walton a écrit : On 4/27/13 4:06 AM, David Bruant wrote: I believe too this is doomed to happen. Firefox is under a huge struggle to remove as much as sync IO code as possible as part of the Snappy effort [1]. This is a completely misleading comparison. Firefox's

Re: [rust-dev] Update on I/O progress

2013-04-26 Thread Gábor Lehel
Probably a crazy idea, but would it be possible to do something where if the parent task launches a child task, and then does nothing else except wait for the child to finish, it's not actually implemented by launching a new task, but instead by calling the closure in place? It seems semantically

Re: [rust-dev] Update on I/O progress

2013-04-26 Thread Patrick Walton
On 4/26/13 6:53 AM, Gábor Lehel wrote: Probably a crazy idea, but would it be possible to do something where if the parent task launches a child task, and then does nothing else except wait for the child to finish, it's not actually implemented by launching a new task, but instead by calling the

Re: [rust-dev] Update on I/O progress

2013-04-26 Thread Patrick Walton
On 4/25/13 10:42 PM, james wrote: If not, I wonder if there should be more effort on the AIO layer first, and then consider a synchronous shim on top. That's exactly what's being done. Patrick ___ Rust-dev mailing list Rust-dev@mozilla.org

Re: [rust-dev] Update on I/O progress

2013-04-26 Thread Graydon Hoare
On 25/04/2013 10:42 PM, james wrote: This always concerns me - the design will tend towards having state on the stack etc and 'pull' processing rather than focussing on a clean design for a chain of state machines that handle 'push' of data coming from AIO completion events. I think push and

Re: [rust-dev] Update on I/O progress

2013-04-26 Thread Patrick Walton
So here are my ideas. They are very worse-is-better at this point. * Failing to perform a basic I/O operation should result in a call to the `io_error` condition (default behavior: fail task) and should set an internal flag saying that this stream is dead. Stream constructors like `open`

Re: [rust-dev] Update on I/O progress

2013-04-26 Thread Brian Anderson
On 04/26/2013 09:04 AM, Patrick Walton wrote: On 4/26/13 6:53 AM, Gábor Lehel wrote: Probably a crazy idea, but would it be possible to do something where if the parent task launches a child task, and then does nothing else except wait for the child to finish, it's not actually implemented by

Re: [rust-dev] Update on I/O progress

2013-04-26 Thread Brian Anderson
On 04/25/2013 06:09 PM, Vadim wrote: What if Result implemented the Drop trait and fail()'ed in drop() unless it's been disarmed by calling some method (get_err?) which indicates that the error has been observed? I've had a few bits of code recently that called for this sort of pattern

Re: [rust-dev] Update on I/O progress

2013-04-26 Thread Vadim
I share your concern. It's likely that Rust tasks will never scale as well as Erlang and I suspect that we will ultimately want an async API that does not impose the expense of 1 task per I/O stream. I hope though that the current approach will scale 'pretty well'. Is there something

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Nathan Myers
On 04/24/2013 03:38 PM, Brian Anderson wrote: ## String encoding I have not yet put much thought about how to deal with string encoding and decoding. The existing `io` module simply has Reader and Writer extension traits that add a number of methods like `read_str`, etc. I think this is the

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Daniel Micay
On Thu, Apr 25, 2013 at 8:25 AM, Nathan Myers n...@cantrip.org wrote: On 04/24/2013 03:38 PM, Brian Anderson wrote: ## String encoding I have not yet put much thought about how to deal with string encoding and decoding. The existing `io` module simply has Reader and Writer extension traits

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Gábor Lehel
You write: So what actually happens if `new` encounters an error? To understand that it's important to know that what `new` returns is not a `File` but an `OptionFile`. If the file does not open, and the condition is handled, then `new` will simply return `None`. Because there is an

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Brian Anderson
On 04/25/2013 05:25 AM, Nathan Myers wrote: On 04/24/2013 03:38 PM, Brian Anderson wrote: ## String encoding I have not yet put much thought about how to deal with string encoding and decoding. The existing `io` module simply has Reader and Writer extension traits that add a number of

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Graydon Hoare
On 25/04/2013 7:18 AM, Gábor Lehel wrote: I'm an outsider, so apologies if I'm off the right track. But this sounds wrong to me. Isn't the point of condition handlers to /handle the condition/? Meaning resolve the problem, so that processing can continue? Here as far as I can tell, they're only

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Brian Anderson
On 04/25/2013 07:12 AM, Daniel Micay wrote: On Thu, Apr 25, 2013 at 8:25 AM, Nathan Myers n...@cantrip.org wrote: On 04/24/2013 03:38 PM, Brian Anderson wrote: ## String encoding I have not yet put much thought about how to deal with string encoding and decoding. The existing `io` module

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Brian Anderson
On 04/25/2013 12:09 PM, Graydon Hoare wrote: On 25/04/2013 7:18 AM, Gábor Lehel wrote: I'm an outsider, so apologies if I'm off the right track. But this sounds wrong to me. Isn't the point of condition handlers to /handle the condition/? Meaning resolve the problem, so that processing can

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Patrick Walton
On 4/25/13 3:46 PM, Brian Anderson wrote: enum FileStream { RtFileStream(rtio::FileStream), // Not an fd_t, a boxed wrapper around uv/rt handles FileSimulation(~Stream), NullFileStream } With this factoring though you are going to be duplicating a lot of logic for FileStream,

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Brian Anderson
On 04/25/2013 03:46 PM, Brian Anderson wrote: On 04/25/2013 12:09 PM, Graydon Hoare wrote: and so forth on other types. Then your file conditions would look like: condition! { no_such_file : Path - FileStream; } and condition! { write_error : (libc::fd_t, libc::errno_t) - FileStream;

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Graydon Hoare
On 25/04/2013 4:02 PM, Brian Anderson wrote: The main TCP implementation, `TcpStream for ~RtTcpStream`, will delegate to whatever boxed RtTcpStream it is given. Because TcpStream is responsible for raising conditions, then you can configure the scheduler to provide you with either uv-based or

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Gábor Lehel
On Thu, Apr 25, 2013 at 10:00 PM, Brian Anderson bander...@mozilla.comwrote: On 04/25/2013 07:18 AM, Gábor Lehel wrote: Whether to respond to failure by failing the task or by indicating it in the return value seems like it would be better handled by having separate functions for each. In

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Brian Anderson
On 04/25/2013 05:08 PM, Gábor Lehel wrote: On Thu, Apr 25, 2013 at 10:00 PM, Brian Anderson bander...@mozilla.com mailto:bander...@mozilla.com wrote: On 04/25/2013 07:18 AM, Gábor Lehel wrote: Whether to respond to failure by failing the task or by indicating it in the

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Patrick Walton
On 4/25/13 5:29 PM, Brian Anderson wrote: It's almost taboo here, but we could consider adding catchable exceptions. I would be in favor of catchable exceptions, for what it's worth. Patrick ___ Rust-dev mailing list Rust-dev@mozilla.org

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Vadim
What if Result implemented the Drop trait and fail()'ed in drop() unless it's been disarmed by calling some method (get_err?) which indicates that the error has been observed? On Thu, Apr 25, 2013 at 5:30 PM, Patrick Walton pwal...@mozilla.com wrote: On 4/25/13 5:29 PM, Brian Anderson wrote:

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread james
On 24/04/2013 23:38, Brian Anderson wrote: What I am focused on now is the design of a synchronous API for sending and receiving streams of bytes, along with an implementation built on an internal *asynchronous* I/O interface attached to the scheduler event loop. This includes reading and