On Mon, May 21, 2012 at 2:03 PM, Bruno Jouhier <[email protected]> wrote:
> luvmonkey really helped me but I got a bit bored playing with the timer > towards the end. Will be nice to have more APIs > Yeah, that should get better as soon as I find time to work on it again. > Regarding the backpressure issue I forgot to mention that there is no > explicit backpressure handling logic in the pump loop itself (it is just a > loop). It works sorta "naturally" thanks to the events dispatched by the > event loop. This means that it will work with arbitrary topologies (for > example several inputs being joined into one output, one input being > dispatched to several outputs, etc). It also works with arbitrary logic in > the middle (complex transforms, etc.). So it is a decoupled and flexible > solution. I also found callback based backpressure very easy while working on proxy node streams between machines (https://github.com/c9/vfs/tree/master/socket) But I'm afraid this won't be fast enough for many people. Not with the default behavior of not calling the callback till the drain event happens. I guess there could be a config option in the wrapping code that looks for some low-water mark and calls the callback before the buffer is completely empty. There is nothing intrinsic to the callback based approach to backpressure that forces waiting for a full drain. > On Monday, May 21, 2012 4:20:56 PM UTC+2, Tim Caswell wrote: >> >> Nice work! Now I've got more motivation to get LuvMonkey into a more >> usable state. >> >> On Sat, May 19, 2012 at 4:06 PM, Bruno Jouhier <[email protected]>wrote: >> >>> I'm not sure I get it but I'll try to answer. >>> >>> What I'm describing in the post is how logic can be expressed with >>> generators rather than callbacks. I'm assuming that the low level calls are >>> callback-style. So, there is no reference to any specific I/O library >>> and/or to back pressure. >>> >>> The back pressure problem is a problem that I'm handling in streamline's >>> streams module. And I'm handling it with a simple pair of async calls: >>> stream.read(cb) and stream.write(cb, buffer) that are small wrappers around >>> node streams. >>> >>> The read call pauses and resumes the underlying stream based on some >>> configurable high/low mark buffering limits (you can set them to 0 but then >>> the stream will pause every time it needs to buffer a chunk). >>> >>> The write call deals with the drain event under the hood. If the lower >>> level call write call returns true, the callback is called immediately >>> (streamline trampolines so there is no risk of stack overflow in callback >>> mode). If it returns false, the callback is triggered by the drain event. >>> >>> Pump loops can be written as: >>> >>> while (data = input.read(_)) >>> output.write(_, data); >>> >>> In callback mode, streamline transforms this loop into something like: >>> >>> (function loop() { >>> input.read(function(err, data) { >>> if (err) return cb(err); >>> if (data) >>> output.write(function(err) { >>> if (err) return cb(err); >>> loop(): >>> }, data); >>> else cb(); >>> }); >>> })(); >>> >>> In generators mode it transforms it into: >>> >>> while (data = yield input.read(_)) >>> yield output.write(_, data) >>> >>> The code looks very different but it execute just like the callback code >>> above. The input.read and output.write calls will use "invoke" and >>> callbacks to interact with the underlying node streams APIs. The run loop >>> that I've given in my post will exit at spot (c) every time a callback is >>> pending and the callback will reactivate it by calling resume; so, even >>> though the code does not look async and callback driven it is actually >>> completely async and callback driven. >>> >>> With this pump loop, backpressure happens naturally when the two calls >>> are combined together. If the output is slower than the input when pumping >>> from an input stream to an output stream, the write call will start to wait >>> for drain events. This will naturally stop the pump loop. The input stream >>> will continue to receive data events but it will just buffer because read >>> won't be called by the pump loop any more. When buffering goes over the >>> high mark, the input stream will be paused. Then, at some point, the output >>> stream will receive a drain event. It will call its callback, which will >>> resume the pump loop. Read will be called and will get data that has been >>> buffered. The input stream will be resumed when buffering goes below the >>> low mark, etc., etc. If the drain event comes before the input reaches the >>> high water mark, the loop will be resumed and the input stream won't be >>> paused, which is what we want. >>> >>> So, even though the pump loop is written as a simple while (data = >>> input.read(_)) output.write(_, data), it does handle the back pressure. >>> >>> Bruno >>> >>> >>> On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote: >>>> >>>> How do you handle back pressure? >>>> >>>> On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote: >>>> >>>> Yes, I fixed it. Thanks. >>>> >>>> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote: >>>>> >>>>> On 5/19/2012 6:20 AM, Bruno Jouhier wrote: >>>>> > http://bjouhier.wordpress.com/****2012/05/18/asynchronous-**javasc** >>>>> ript-with-generators-an-**experi**ment/<http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/> >>>>> >>>>> shouldn't that be print(num) not print(n) >>>>> >>>>> >>>> -- >>>> Job Board: http://jobs.nodejs.org/ >>>> Posting guidelines: https://github.com/joyent/**node** >>>> /wiki/Mailing-List-**Posting-**Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>>> You received this message because you are subscribed to the Google >>>> Groups "nodejs" group. >>>> To post to this group, send email to [email protected] >>>> To unsubscribe from this group, send email to >>>> nodejs+unsubscribe@**googlegroup**s.com<[email protected]> >>>> For more options, visit this group at >>>> http://groups.google.com/**group**/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>>> >>>> >>>> >>> On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote: >>>> >>>> How do you handle back pressure? >>>> >>>> On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote: >>>> >>>> Yes, I fixed it. Thanks. >>>> >>>> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote: >>>>> >>>>> On 5/19/2012 6:20 AM, Bruno Jouhier wrote: >>>>> > http://bjouhier.wordpress.com/****2012/05/18/asynchronous-**javasc** >>>>> ript-with-generators-an-**experi**ment/<http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/> >>>>> >>>>> shouldn't that be print(num) not print(n) >>>>> >>>>> >>>> -- >>>> Job Board: http://jobs.nodejs.org/ >>>> Posting guidelines: https://github.com/joyent/**node** >>>> /wiki/Mailing-List-**Posting-**Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>>> You received this message because you are subscribed to the Google >>>> Groups "nodejs" group. >>>> To post to this group, send email to [email protected] >>>> To unsubscribe from this group, send email to >>>> nodejs+unsubscribe@**googlegroup**s.com<[email protected]> >>>> For more options, visit this group at >>>> http://groups.google.com/**group**/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>>> >>>> >>>> >>> On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote: >>>> >>>> How do you handle back pressure? >>>> >>>> On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote: >>>> >>>> Yes, I fixed it. Thanks. >>>> >>>> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote: >>>>> >>>>> On 5/19/2012 6:20 AM, Bruno Jouhier wrote: >>>>> > http://bjouhier.wordpress.com/****2012/05/18/asynchronous-**javasc** >>>>> ript-with-generators-an-**experi**ment/<http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/> >>>>> >>>>> shouldn't that be print(num) not print(n) >>>>> >>>>> >>>> -- >>>> Job Board: http://jobs.nodejs.org/ >>>> Posting guidelines: https://github.com/joyent/**node** >>>> /wiki/Mailing-List-**Posting-**Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>>> You received this message because you are subscribed to the Google >>>> Groups "nodejs" group. >>>> To post to this group, send email to [email protected] >>>> To unsubscribe from this group, send email to >>>> nodejs+unsubscribe@**googlegroup**s.com<[email protected]> >>>> For more options, visit this group at >>>> http://groups.google.com/**group**/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>>> >>>> >>>> >>> On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote: >>>> >>>> How do you handle back pressure? >>>> >>>> On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote: >>>> >>>> Yes, I fixed it. Thanks. >>>> >>>> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote: >>>>> >>>>> On 5/19/2012 6:20 AM, Bruno Jouhier wrote: >>>>> > http://bjouhier.wordpress.com/****2012/05/18/asynchronous-**javasc** >>>>> ript-with-generators-an-**experi**ment/<http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/> >>>>> >>>>> shouldn't that be print(num) not print(n) >>>>> >>>>> >>>> -- >>>> Job Board: http://jobs.nodejs.org/ >>>> Posting guidelines: https://github.com/joyent/**node** >>>> /wiki/Mailing-List-**Posting-**Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>>> You received this message because you are subscribed to the Google >>>> Groups "nodejs" group. >>>> To post to this group, send email to [email protected] >>>> To unsubscribe from this group, send email to >>>> nodejs+unsubscribe@**googlegroup**s.com<[email protected]> >>>> For more options, visit this group at >>>> http://groups.google.com/**group**/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>>> >>>> >>>> -- >>> Job Board: http://jobs.nodejs.org/ >>> Posting guidelines: https://github.com/joyent/**node/wiki/Mailing-List-* >>> *Posting-Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>> You received this message because you are subscribed to the Google >>> Groups "nodejs" group. >>> To post to this group, send email to [email protected] >>> To unsubscribe from this group, send email to >>> nodejs+unsubscribe@**googlegroups.com<nodejs%[email protected]> >>> For more options, visit this group at >>> http://groups.google.com/**group/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>> >> >> -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
