One thing I'm not getting is what coros have to do in all this. I can use streamline in 3 modes (callbacks, fibers and generators) which are equivalent (the streamline source behaves the same, even though it gets compiled differently). The last 2 modes use some form of coroutines but the first one does not. So when I use the streamline streams module in callbacks mode, there is no coro involved at all, it is pure JS with callbacks.
On Monday, May 21, 2012 9:18:07 PM UTC+2, Mikeal Rogers wrote: > > > On May 21, 2012, at May 21, 201212:03 PM, Bruno Jouhier wrote: > > luvmonkey really helped me but I got a bit bored playing with the timer > towards the end. Will be nice to have more APIs > > Regarding the backpressure issue I forgot to mention that there is no > explicit backpressure handling logic in the pump loop itself (it is just a > loop). It works sorta "naturally" thanks to the events dispatched by the > event loop. This means that it will work with arbitrary topologies (for > example several inputs being joined into one output, one input being > dispatched to several outputs, etc). It also works with arbitrary logic in > the middle (complex transforms, etc.). So it is a decoupled and flexible > solution. > > > In theory, but not in practice. > > As we've already seen with streams you can't generalize the back pressure > logic for multi stream outputs. Everyone who is actually doing this in > production has their own application specific versions of pipe() that > decide when and if the inputs should actually be paused. > > I'm skeptical you're actually handling the back pressure case properly. > coro systems in Ruby and Python don't really handle it either and it's a > problem with generators that we've spent a lot of time discussing and > exploring. This debate, in javascript, goes all the way back to the JSGI > discussions before node had streams and the eJSGI work that @isaacs did > which also pre-dates streams. > > Back pressure from file descriptors, sockets in particular, is one of the > most important parts of node streams and is crucial to concurrent > performance with mobile client. If you're not prioritizing it then I can't > take this project seriously. If you are then I'd be very interested to see > how you tackle the issues we've explored already in the work that pre-dated > streams. > > Expressing logic as generators rather than callbacks is a decent academic > experiment but it's not tackling the hard problems we tackle with streams, > nor are the comparisons as compelling since streams being piped together > doesn't expose any callback indentation. Ignoring these cases or contriving > examples with file descriptors that aren't using streams is a very thin > strawman. > > > > On Monday, May 21, 2012 4:20:56 PM UTC+2, Tim Caswell wrote: >> >> Nice work! Now I've got more motivation to get LuvMonkey into a more >> usable state. >> >> On Sat, May 19, 2012 at 4:06 PM, Bruno Jouhier <[email protected]>wrote: >> >>> I'm not sure I get it but I'll try to answer. >>> >>> What I'm describing in the post is how logic can be expressed with >>> generators rather than callbacks. I'm assuming that the low level calls are >>> callback-style. So, there is no reference to any specific I/O library >>> and/or to back pressure. >>> >>> The back pressure problem is a problem that I'm handling in streamline's >>> streams module. And I'm handling it with a simple pair of async calls: >>> stream.read(cb) and stream.write(cb, buffer) that are small wrappers around >>> node streams. >>> >>> The read call pauses and resumes the underlying stream based on some >>> configurable high/low mark buffering limits (you can set them to 0 but then >>> the stream will pause every time it needs to buffer a chunk). >>> >>> The write call deals with the drain event under the hood. If the lower >>> level call write call returns true, the callback is called immediately >>> (streamline trampolines so there is no risk of stack overflow in callback >>> mode). If it returns false, the callback is triggered by the drain event. >>> >>> Pump loops can be written as: >>> >>> while (data = input.read(_)) >>> output.write(_, data); >>> >>> In callback mode, streamline transforms this loop into something like: >>> >>> (function loop() { >>> input.read(function(err, data) { >>> if (err) return cb(err); >>> if (data) >>> output.write(function(err) { >>> if (err) return cb(err); >>> loop(): >>> }, data); >>> else cb(); >>> }); >>> })(); >>> >>> In generators mode it transforms it into: >>> >>> while (data = yield input.read(_)) >>> yield output.write(_, data) >>> >>> The code looks very different but it execute just like the callback code >>> above. The input.read and output.write calls will use "invoke" and >>> callbacks to interact with the underlying node streams APIs. The run loop >>> that I've given in my post will exit at spot (c) every time a callback is >>> pending and the callback will reactivate it by calling resume; so, even >>> though the code does not look async and callback driven it is actually >>> completely async and callback driven. >>> >>> With this pump loop, backpressure happens naturally when the two calls >>> are combined together. If the output is slower than the input when pumping >>> from an input stream to an output stream, the write call will start to wait >>> for drain events. This will naturally stop the pump loop. The input stream >>> will continue to receive data events but it will just buffer because read >>> won't be called by the pump loop any more. When buffering goes over the >>> high mark, the input stream will be paused. Then, at some point, the output >>> stream will receive a drain event. It will call its callback, which will >>> resume the pump loop. Read will be called and will get data that has been >>> buffered. The input stream will be resumed when buffering goes below the >>> low mark, etc., etc. If the drain event comes before the input reaches the >>> high water mark, the loop will be resumed and the input stream won't be >>> paused, which is what we want. >>> >>> So, even though the pump loop is written as a simple while (data = >>> input.read(_)) output.write(_, data), it does handle the back pressure. >>> >>> Bruno >>> >>> >>> On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote: >>>> >>>> How do you handle back pressure? >>>> >>>> On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote: >>>> >>>> Yes, I fixed it. Thanks. >>>> >>>> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote: >>>>> >>>>> On 5/19/2012 6:20 AM, Bruno Jouhier wrote: >>>>> > http://bjouhier.wordpress.com/**2012/05/18/asynchronous-** >>>>> javascript-with-generators-an-**experiment/<http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/> >>>>> >>>>> >>>>> shouldn't that be print(num) not print(n) >>>>> >>>>> >>>> -- >>>> Job Board: http://jobs.nodejs.org/ >>>> Posting guidelines: https://github.com/joyent/**node/wiki/Mailing-List- >>>> **Posting-Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>>> You received this message because you are subscribed to the Google >>>> Groups "nodejs" group. >>>> To post to this group, send email to [email protected] >>>> To unsubscribe from this group, send email to >>>> nodejs+unsubscribe@**googlegroups.com<[email protected]> >>>> For more options, visit this group at >>>> http://groups.google.com/**group/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>>> >>>> >>>> >>> On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote: >>>> >>>> How do you handle back pressure? >>>> >>>> On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote: >>>> >>>> Yes, I fixed it. Thanks. >>>> >>>> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote: >>>>> >>>>> On 5/19/2012 6:20 AM, Bruno Jouhier wrote: >>>>> > http://bjouhier.wordpress.com/**2012/05/18/asynchronous-** >>>>> javascript-with-generators-an-**experiment/<http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/> >>>>> >>>>> >>>>> shouldn't that be print(num) not print(n) >>>>> >>>>> >>>> -- >>>> Job Board: http://jobs.nodejs.org/ >>>> Posting guidelines: https://github.com/joyent/**node/wiki/Mailing-List- >>>> **Posting-Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>>> You received this message because you are subscribed to the Google >>>> Groups "nodejs" group. >>>> To post to this group, send email to [email protected] >>>> To unsubscribe from this group, send email to >>>> nodejs+unsubscribe@**googlegroups.com<[email protected]> >>>> For more options, visit this group at >>>> http://groups.google.com/**group/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>>> >>>> >>>> >>> On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote: >>>> >>>> How do you handle back pressure? >>>> >>>> On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote: >>>> >>>> Yes, I fixed it. Thanks. >>>> >>>> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote: >>>>> >>>>> On 5/19/2012 6:20 AM, Bruno Jouhier wrote: >>>>> > http://bjouhier.wordpress.com/**2012/05/18/asynchronous-** >>>>> javascript-with-generators-an-**experiment/<http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/> >>>>> >>>>> >>>>> shouldn't that be print(num) not print(n) >>>>> >>>>> >>>> -- >>>> Job Board: http://jobs.nodejs.org/ >>>> Posting guidelines: https://github.com/joyent/**node/wiki/Mailing-List- >>>> **Posting-Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>>> You received this message because you are subscribed to the Google >>>> Groups "nodejs" group. >>>> To post to this group, send email to [email protected] >>>> To unsubscribe from this group, send email to >>>> nodejs+unsubscribe@**googlegroups.com<[email protected]> >>>> For more options, visit this group at >>>> http://groups.google.com/**group/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>>> >>>> >>>> >>> On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote: >>>> >>>> How do you handle back pressure? >>>> >>>> On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote: >>>> >>>> Yes, I fixed it. Thanks. >>>> >>>> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote: >>>>> >>>>> On 5/19/2012 6:20 AM, Bruno Jouhier wrote: >>>>> > http://bjouhier.wordpress.com/**2012/05/18/asynchronous-** >>>>> javascript-with-generators-an-**experiment/<http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/> >>>>> >>>>> >>>>> shouldn't that be print(num) not print(n) >>>>> >>>>> >>>> -- >>>> Job Board: http://jobs.nodejs.org/ >>>> Posting guidelines: https://github.com/joyent/**node/wiki/Mailing-List- >>>> **Posting-Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>>> You received this message because you are subscribed to the Google >>>> Groups "nodejs" group. >>>> To post to this group, send email to [email protected] >>>> To unsubscribe from this group, send email to >>>> nodejs+unsubscribe@**googlegroups.com<[email protected]> >>>> For more options, visit this group at >>>> http://groups.google.com/**group/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>>> >>>> >>>> >>> -- >>> Job Board: http://jobs.nodejs.org/ >>> Posting guidelines: >>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >>> You received this message because you are subscribed to the Google >>> Groups "nodejs" group. >>> To post to this group, send email to [email protected] >>> To unsubscribe from this group, send email to >>> [email protected] >>> For more options, visit this group at >>> http://groups.google.com/group/nodejs?hl=en?hl=en >>> >> >> > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > > > -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
