On Monday, May 21, 2012 10:08:03 PM UTC+2, Bruno Jouhier wrote: > > I agree that nothing will beat util.pump when it comes to pure speed > pumping one stream into another. > I meant stream.pipe
> > The lowmark enhancement you're talking about should also benefit util.pump > because it connects drain directly to resume. So it waits for full drain > too (I'm looking at util.js in the master branch). > > On Monday, May 21, 2012 9:16:06 PM UTC+2, Tim Caswell wrote: >> >> >> On Mon, May 21, 2012 at 2:03 PM, Bruno Jouhier <[email protected]>wrote: >> >>> luvmonkey really helped me but I got a bit bored playing with the timer >>> towards the end. Will be nice to have more APIs >>> >> >> Yeah, that should get better as soon as I find time to work on it again. >> >> >>> Regarding the backpressure issue I forgot to mention that there is no >>> explicit backpressure handling logic in the pump loop itself (it is just a >>> loop). It works sorta "naturally" thanks to the events dispatched by the >>> event loop. This means that it will work with arbitrary topologies (for >>> example several inputs being joined into one output, one input being >>> dispatched to several outputs, etc). It also works with arbitrary logic in >>> the middle (complex transforms, etc.). So it is a decoupled and flexible >>> solution. >> >> >> I also found callback based backpressure very easy while working on proxy >> node streams between machines ( >> https://github.com/c9/vfs/tree/master/socket) But I'm afraid this won't >> be fast enough for many people. Not with the default behavior of not >> calling the callback till the drain event happens. I guess there could be >> a config option in the wrapping code that looks for some low-water mark and >> calls the callback before the buffer is completely empty. There is nothing >> intrinsic to the callback based approach to backpressure that forces >> waiting for a full drain. >> >> >> >>> On Monday, May 21, 2012 4:20:56 PM UTC+2, Tim Caswell wrote: >>>> >>>> Nice work! Now I've got more motivation to get LuvMonkey into a more >>>> usable state. >>>> >>>> On Sat, May 19, 2012 at 4:06 PM, Bruno Jouhier <[email protected]>wrote: >>>> >>>>> I'm not sure I get it but I'll try to answer. >>>>> >>>>> What I'm describing in the post is how logic can be expressed with >>>>> generators rather than callbacks. I'm assuming that the low level calls >>>>> are >>>>> callback-style. So, there is no reference to any specific I/O library >>>>> and/or to back pressure. >>>>> >>>>> The back pressure problem is a problem that I'm handling in >>>>> streamline's streams module. And I'm handling it with a simple pair of >>>>> async calls: stream.read(cb) and stream.write(cb, buffer) that are small >>>>> wrappers around node streams. >>>>> >>>>> The read call pauses and resumes the underlying stream based on some >>>>> configurable high/low mark buffering limits (you can set them to 0 but >>>>> then >>>>> the stream will pause every time it needs to buffer a chunk). >>>>> >>>>> The write call deals with the drain event under the hood. If the lower >>>>> level call write call returns true, the callback is called immediately >>>>> (streamline trampolines so there is no risk of stack overflow in callback >>>>> mode). If it returns false, the callback is triggered by the drain event. >>>>> >>>>> Pump loops can be written as: >>>>> >>>>> while (data = input.read(_)) >>>>> output.write(_, data); >>>>> >>>>> In callback mode, streamline transforms this loop into something like: >>>>> >>>>> (function loop() { >>>>> input.read(function(err, data) { >>>>> if (err) return cb(err); >>>>> if (data) >>>>> output.write(function(err) { >>>>> if (err) return cb(err); >>>>> loop(): >>>>> }, data); >>>>> else cb(); >>>>> }); >>>>> })(); >>>>> >>>>> In generators mode it transforms it into: >>>>> >>>>> while (data = yield input.read(_)) >>>>> yield output.write(_, data) >>>>> >>>>> The code looks very different but it execute just like the callback >>>>> code above. The input.read and output.write calls will use "invoke" and >>>>> callbacks to interact with the underlying node streams APIs. The run loop >>>>> that I've given in my post will exit at spot (c) every time a callback is >>>>> pending and the callback will reactivate it by calling resume; so, even >>>>> though the code does not look async and callback driven it is actually >>>>> completely async and callback driven. >>>>> >>>>> With this pump loop, backpressure happens naturally when the two calls >>>>> are combined together. If the output is slower than the input when >>>>> pumping >>>>> from an input stream to an output stream, the write call will start to >>>>> wait >>>>> for drain events. This will naturally stop the pump loop. The input >>>>> stream >>>>> will continue to receive data events but it will just buffer because read >>>>> won't be called by the pump loop any more. When buffering goes over the >>>>> high mark, the input stream will be paused. Then, at some point, the >>>>> output >>>>> stream will receive a drain event. It will call its callback, which will >>>>> resume the pump loop. Read will be called and will get data that has been >>>>> buffered. The input stream will be resumed when buffering goes below the >>>>> low mark, etc., etc. If the drain event comes before the input reaches >>>>> the >>>>> high water mark, the loop will be resumed and the input stream won't be >>>>> paused, which is what we want. >>>>> >>>>> So, even though the pump loop is written as a simple while (data = >>>>> input.read(_)) output.write(_, data), it does handle the back pressure. >>>>> >>>>> Bruno >>>>> >>>>> >>>>> On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote: >>>>>> >>>>>> How do you handle back pressure? >>>>>> >>>>>> On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote: >>>>>> >>>>>> Yes, I fixed it. Thanks. >>>>>> >>>>>> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote: >>>>>>> >>>>>>> On 5/19/2012 6:20 AM, Bruno Jouhier wrote: >>>>>>> > http://bjouhier.wordpress.com/****2012/05/18/asynchronous-**javasc >>>>>>> **ript-with-generators-an-**experi**ment/<http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/> >>>>>>> >>>>>>> >>>>>>> shouldn't that be print(num) not print(n) >>>>>>> >>>>>>> >>>>>> -- >>>>>> Job Board: http://jobs.nodejs.org/ >>>>>> Posting guidelines: https://github.com/joyent/**node** >>>>>> /wiki/Mailing-List-**Posting-**Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "nodejs" group. >>>>>> To post to this group, send email to [email protected] >>>>>> To unsubscribe from this group, send email to >>>>>> nodejs+unsubscribe@**googlegroup**s.com<[email protected]> >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/**group**/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>>>>> >>>>>> >>>>>> >>>>> On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote: >>>>>> >>>>>> How do you handle back pressure? >>>>>> >>>>>> On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote: >>>>>> >>>>>> Yes, I fixed it. Thanks. >>>>>> >>>>>> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote: >>>>>>> >>>>>>> On 5/19/2012 6:20 AM, Bruno Jouhier wrote: >>>>>>> > http://bjouhier.wordpress.com/****2012/05/18/asynchronous-**javasc >>>>>>> **ript-with-generators-an-**experi**ment/<http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/> >>>>>>> >>>>>>> >>>>>>> shouldn't that be print(num) not print(n) >>>>>>> >>>>>>> >>>>>> -- >>>>>> Job Board: http://jobs.nodejs.org/ >>>>>> Posting guidelines: https://github.com/joyent/**node** >>>>>> /wiki/Mailing-List-**Posting-**Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "nodejs" group. >>>>>> To post to this group, send email to [email protected] >>>>>> To unsubscribe from this group, send email to >>>>>> nodejs+unsubscribe@**googlegroup**s.com<[email protected]> >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/**group**/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>>>>> >>>>>> >>>>>> >>>>> On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote: >>>>>> >>>>>> How do you handle back pressure? >>>>>> >>>>>> On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote: >>>>>> >>>>>> Yes, I fixed it. Thanks. >>>>>> >>>>>> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote: >>>>>>> >>>>>>> On 5/19/2012 6:20 AM, Bruno Jouhier wrote: >>>>>>> > http://bjouhier.wordpress.com/****2012/05/18/asynchronous-**javasc >>>>>>> **ript-with-generators-an-**experi**ment/<http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/> >>>>>>> >>>>>>> >>>>>>> shouldn't that be print(num) not print(n) >>>>>>> >>>>>>> >>>>>> -- >>>>>> Job Board: http://jobs.nodejs.org/ >>>>>> Posting guidelines: https://github.com/joyent/**node** >>>>>> /wiki/Mailing-List-**Posting-**Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "nodejs" group. >>>>>> To post to this group, send email to [email protected] >>>>>> To unsubscribe from this group, send email to >>>>>> nodejs+unsubscribe@**googlegroup**s.com<[email protected]> >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/**group**/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>>>>> >>>>>> >>>>>> >>>>> On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote: >>>>>> >>>>>> How do you handle back pressure? >>>>>> >>>>>> On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote: >>>>>> >>>>>> Yes, I fixed it. Thanks. >>>>>> >>>>>> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote: >>>>>>> >>>>>>> On 5/19/2012 6:20 AM, Bruno Jouhier wrote: >>>>>>> > http://bjouhier.wordpress.com/****2012/05/18/asynchronous-**javasc >>>>>>> **ript-with-generators-an-**experi**ment/<http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/> >>>>>>> >>>>>>> >>>>>>> shouldn't that be print(num) not print(n) >>>>>>> >>>>>>> >>>>>> -- >>>>>> Job Board: http://jobs.nodejs.org/ >>>>>> Posting guidelines: https://github.com/joyent/**node** >>>>>> /wiki/Mailing-List-**Posting-**Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "nodejs" group. >>>>>> To post to this group, send email to [email protected] >>>>>> To unsubscribe from this group, send email to >>>>>> nodejs+unsubscribe@**googlegroup**s.com<[email protected]> >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/**group**/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>>>>> >>>>>> >>>>>> -- >>>>> Job Board: http://jobs.nodejs.org/ >>>>> Posting guidelines: https://github.com/joyent/** >>>>> node/wiki/Mailing-List-**Posting-Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>>>> You received this message because you are subscribed to the Google >>>>> Groups "nodejs" group. >>>>> To post to this group, send email to [email protected] >>>>> To unsubscribe from this group, send email to >>>>> nodejs+unsubscribe@**googlegroups.com<nodejs%[email protected]> >>>>> For more options, visit this group at >>>>> http://groups.google.com/**group/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>>>> >>>> >>>> -- >>> Job Board: http://jobs.nodejs.org/ >>> Posting guidelines: >>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >>> You received this message because you are subscribed to the Google >>> Groups "nodejs" group. >>> To post to this group, send email to [email protected] >>> To unsubscribe from this group, send email to >>> [email protected] >>> For more options, visit this group at >>> http://groups.google.com/group/nodejs?hl=en?hl=en >>> >> >> -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
