On Mon, May 21, 2012 at 2:18 PM, Mikeal Rogers <[email protected]>wrote:
> > On May 21, 2012, at May 21, 201212:03 PM, Bruno Jouhier wrote: > > luvmonkey really helped me but I got a bit bored playing with the timer > towards the end. Will be nice to have more APIs > > Regarding the backpressure issue I forgot to mention that there is no > explicit backpressure handling logic in the pump loop itself (it is just a > loop). It works sorta "naturally" thanks to the events dispatched by the > event loop. This means that it will work with arbitrary topologies (for > example several inputs being joined into one output, one input being > dispatched to several outputs, etc). It also works with arbitrary logic in > the middle (complex transforms, etc.). So it is a decoupled and flexible > solution. > > > In theory, but not in practice. > > As we've already seen with streams you can't generalize the back pressure > logic for multi stream outputs. Everyone who is actually doing this in > production has their own application specific versions of pipe() that > decide when and if the inputs should actually be paused. > > I'm skeptical you're actually handling the back pressure case properly. > coro systems in Ruby and Python don't really handle it either and it's a > problem with generators that we've spent a lot of time discussing and > exploring. This debate, in javascript, goes all the way back to the JSGI > discussions before node had streams and the eJSGI work that @isaacs did > which also pre-dates streams. > > Back pressure from file descriptors, sockets in particular, is one of the > most important parts of node streams and is crucial to concurrent > performance with mobile client. If you're not prioritizing it then I can't > take this project seriously. If you are then I'd be very interested to see > how you tackle the issues we've explored already in the work that pre-dated > streams. > > Expressing logic as generators rather than callbacks is a decent academic > experiment but it's not tackling the hard problems we tackle with streams, > nor are the comparisons as compelling since streams being piped together > doesn't expose any callback indentation. Ignoring these cases or contriving > examples with file descriptors that aren't using streams is a very thin > strawman. > So, to see if I understand Mikeal's concern, let me try to paraphrase it: "Streams are hard, we've worked on this for years and still haven't gotten it right. There is no way it's as easy as you say it is so therefore this isn't going to work." It's ok to be skeptical. I was in some of those early meetings in Oakland where we drew streams on a whiteboard and planned APIs. I know that streams are hard to abstract properly. I've recently had to find a way to implement streams on top of an rpc system with proper backpressure and realized that the current pipe API cannot work generally, even for many simple cases. How about instead we take Bruno's efforts at face value and not try to read too much into intentions. There is some neat technology here and he's giving it away for free and sharing with the community. I will probably never use streamline in any library I write, but I still learn from the project. Also as a disclaimer, I asked Bruno to write a generator backend that can run on LuvMonkey because I'm interested to see how it will run. I still hold my position that newbies who have trouble understanding callback based code shouldn't reach for streamline till they first master the basics. But for experienced developers who just want to write real apps for real people to use (as opposed to libraries used only by other developers), I think streamline can be a great tool to get the job done. Let'st try to keep an open mind and be welcoming to people who share their hard work with us. > > > On Monday, May 21, 2012 4:20:56 PM UTC+2, Tim Caswell wrote: >> >> Nice work! Now I've got more motivation to get LuvMonkey into a more >> usable state. >> >> On Sat, May 19, 2012 at 4:06 PM, Bruno Jouhier <[email protected]>wrote: >> >>> I'm not sure I get it but I'll try to answer. >>> >>> What I'm describing in the post is how logic can be expressed with >>> generators rather than callbacks. I'm assuming that the low level calls are >>> callback-style. So, there is no reference to any specific I/O library >>> and/or to back pressure. >>> >>> The back pressure problem is a problem that I'm handling in streamline's >>> streams module. And I'm handling it with a simple pair of async calls: >>> stream.read(cb) and stream.write(cb, buffer) that are small wrappers around >>> node streams. >>> >>> The read call pauses and resumes the underlying stream based on some >>> configurable high/low mark buffering limits (you can set them to 0 but then >>> the stream will pause every time it needs to buffer a chunk). >>> >>> The write call deals with the drain event under the hood. If the lower >>> level call write call returns true, the callback is called immediately >>> (streamline trampolines so there is no risk of stack overflow in callback >>> mode). If it returns false, the callback is triggered by the drain event. >>> >>> Pump loops can be written as: >>> >>> while (data = input.read(_)) >>> output.write(_, data); >>> >>> In callback mode, streamline transforms this loop into something like: >>> >>> (function loop() { >>> input.read(function(err, data) { >>> if (err) return cb(err); >>> if (data) >>> output.write(function(err) { >>> if (err) return cb(err); >>> loop(): >>> }, data); >>> else cb(); >>> }); >>> })(); >>> >>> In generators mode it transforms it into: >>> >>> while (data = yield input.read(_)) >>> yield output.write(_, data) >>> >>> The code looks very different but it execute just like the callback code >>> above. The input.read and output.write calls will use "invoke" and >>> callbacks to interact with the underlying node streams APIs. The run loop >>> that I've given in my post will exit at spot (c) every time a callback is >>> pending and the callback will reactivate it by calling resume; so, even >>> though the code does not look async and callback driven it is actually >>> completely async and callback driven. >>> >>> With this pump loop, backpressure happens naturally when the two calls >>> are combined together. If the output is slower than the input when pumping >>> from an input stream to an output stream, the write call will start to wait >>> for drain events. This will naturally stop the pump loop. The input stream >>> will continue to receive data events but it will just buffer because read >>> won't be called by the pump loop any more. When buffering goes over the >>> high mark, the input stream will be paused. Then, at some point, the output >>> stream will receive a drain event. It will call its callback, which will >>> resume the pump loop. Read will be called and will get data that has been >>> buffered. The input stream will be resumed when buffering goes below the >>> low mark, etc., etc. If the drain event comes before the input reaches the >>> high water mark, the loop will be resumed and the input stream won't be >>> paused, which is what we want. >>> >>> So, even though the pump loop is written as a simple while (data = >>> input.read(_)) output.write(_, data), it does handle the back pressure. >>> >>> Bruno >>> >>> >>> On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote: >>>> >>>> How do you handle back pressure? >>>> >>>> On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote: >>>> >>>> Yes, I fixed it. Thanks. >>>> >>>> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote: >>>>> >>>>> On 5/19/2012 6:20 AM, Bruno Jouhier wrote: >>>>> > http://bjouhier.wordpress.com/****2012/05/18/asynchronous-**javasc** >>>>> ript-with-generators-an-**experi**ment/<http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/> >>>>> >>>>> shouldn't that be print(num) not print(n) >>>>> >>>>> >>>> -- >>>> Job Board: http://jobs.nodejs.org/ >>>> Posting guidelines: https://github.com/joyent/**node** >>>> /wiki/Mailing-List-**Posting-**Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>>> You received this message because you are subscribed to the Google >>>> Groups "nodejs" group. >>>> To post to this group, send email to [email protected] >>>> To unsubscribe from this group, send email to >>>> nodejs+unsubscribe@**googlegroup**s.com<[email protected]> >>>> For more options, visit this group at >>>> http://groups.google.com/**group**/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>>> >>>> >>>> >>> On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote: >>>> >>>> How do you handle back pressure? >>>> >>>> On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote: >>>> >>>> Yes, I fixed it. Thanks. >>>> >>>> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote: >>>>> >>>>> On 5/19/2012 6:20 AM, Bruno Jouhier wrote: >>>>> > http://bjouhier.wordpress.com/****2012/05/18/asynchronous-**javasc** >>>>> ript-with-generators-an-**experi**ment/<http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/> >>>>> >>>>> shouldn't that be print(num) not print(n) >>>>> >>>>> >>>> -- >>>> Job Board: http://jobs.nodejs.org/ >>>> Posting guidelines: https://github.com/joyent/**node** >>>> /wiki/Mailing-List-**Posting-**Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>>> You received this message because you are subscribed to the Google >>>> Groups "nodejs" group. >>>> To post to this group, send email to [email protected] >>>> To unsubscribe from this group, send email to >>>> nodejs+unsubscribe@**googlegroup**s.com<[email protected]> >>>> For more options, visit this group at >>>> http://groups.google.com/**group**/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>>> >>>> >>>> >>> On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote: >>>> >>>> How do you handle back pressure? >>>> >>>> On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote: >>>> >>>> Yes, I fixed it. Thanks. >>>> >>>> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote: >>>>> >>>>> On 5/19/2012 6:20 AM, Bruno Jouhier wrote: >>>>> > http://bjouhier.wordpress.com/****2012/05/18/asynchronous-**javasc** >>>>> ript-with-generators-an-**experi**ment/<http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/> >>>>> >>>>> shouldn't that be print(num) not print(n) >>>>> >>>>> >>>> -- >>>> Job Board: http://jobs.nodejs.org/ >>>> Posting guidelines: https://github.com/joyent/**node** >>>> /wiki/Mailing-List-**Posting-**Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>>> You received this message because you are subscribed to the Google >>>> Groups "nodejs" group. >>>> To post to this group, send email to [email protected] >>>> To unsubscribe from this group, send email to >>>> nodejs+unsubscribe@**googlegroup**s.com<[email protected]> >>>> For more options, visit this group at >>>> http://groups.google.com/**group**/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>>> >>>> >>>> >>> On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote: >>>> >>>> How do you handle back pressure? >>>> >>>> On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote: >>>> >>>> Yes, I fixed it. Thanks. >>>> >>>> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote: >>>>> >>>>> On 5/19/2012 6:20 AM, Bruno Jouhier wrote: >>>>> > http://bjouhier.wordpress.com/****2012/05/18/asynchronous-**javasc** >>>>> ript-with-generators-an-**experi**ment/<http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/> >>>>> >>>>> shouldn't that be print(num) not print(n) >>>>> >>>>> >>>> -- >>>> Job Board: http://jobs.nodejs.org/ >>>> Posting guidelines: https://github.com/joyent/**node** >>>> /wiki/Mailing-List-**Posting-**Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>>> You received this message because you are subscribed to the Google >>>> Groups "nodejs" group. >>>> To post to this group, send email to [email protected] >>>> To unsubscribe from this group, send email to >>>> nodejs+unsubscribe@**googlegroup**s.com<[email protected]> >>>> For more options, visit this group at >>>> http://groups.google.com/**group**/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>>> >>>> >>>> >>> -- >>> Job Board: http://jobs.nodejs.org/ >>> Posting guidelines: https://github.com/joyent/**node/wiki/Mailing-List-* >>> *Posting-Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>> You received this message because you are subscribed to the Google >>> Groups "nodejs" group. >>> To post to this group, send email to [email protected] >>> To unsubscribe from this group, send email to >>> nodejs+unsubscribe@**googlegroups.com<nodejs%[email protected]> >>> For more options, visit this group at >>> http://groups.google.com/**group/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>> >> >> > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > > > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
