util.pump was deprecated in 0.3. this logic is in Stream.pipe() now. actually, "drain" is an event we get from libuv. it's an implementation detail of libuv that it flushes pending writes before emitting drain.
but, this is actually *more* performant in the case of slow clients. if we emit "drain" before flushing pending writes and those pending writes cause another pause() then we'll suffer another round trip to the source stream, which if it's TCP will be quite long, in order to do another input pause(). my proposal for Stream.pause(size) in 0.9 takes a size parameter that is the size in bytes the stream will hold *before* causing pause() on the input stream. this is useful for cases where the user knows they will have low latency to a source like disc or Redis to answer a question in a callback. when that time is lower than the round trip to the source it's better to just buffer internally and not cause a pause on the input stream. this proposal is still in heavy debate and i'm still convincing @isaacs of a few things. -Mikeal On May 21, 2012, at May 21, 20121:08 PM, Bruno Jouhier wrote: > I agree that nothing will beat util.pump when it comes to pure speed pumping > one stream into another. > > The lowmark enhancement you're talking about should also benefit util.pump > because it connects drain directly to resume. So it waits for full drain too > (I'm looking at util.js in the master branch). > > On Monday, May 21, 2012 9:16:06 PM UTC+2, Tim Caswell wrote: > > On Mon, May 21, 2012 at 2:03 PM, Bruno Jouhier <[email protected]> wrote: > luvmonkey really helped me but I got a bit bored playing with the timer > towards the end. Will be nice to have more APIs > > Yeah, that should get better as soon as I find time to work on it again. > > Regarding the backpressure issue I forgot to mention that there is no > explicit backpressure handling logic in the pump loop itself (it is just a > loop). It works sorta "naturally" thanks to the events dispatched by the > event loop. This means that it will work with arbitrary topologies (for > example several inputs being joined into one output, one input being > dispatched to several outputs, etc). It also works with arbitrary logic in > the middle (complex transforms, etc.). So it is a decoupled and flexible > solution. > > I also found callback based backpressure very easy while working on proxy > node streams between machines (https://github.com/c9/vfs/tree/master/socket) > But I'm afraid this won't be fast enough for many people. Not with the > default behavior of not calling the callback till the drain event happens. I > guess there could be a config option in the wrapping code that looks for some > low-water mark and calls the callback before the buffer is completely empty. > There is nothing intrinsic to the callback based approach to backpressure > that forces waiting for a full drain. > > > On Monday, May 21, 2012 4:20:56 PM UTC+2, Tim Caswell wrote: > Nice work! Now I've got more motivation to get LuvMonkey into a more usable > state. > > On Sat, May 19, 2012 at 4:06 PM, Bruno Jouhier <[email protected]> wrote: > I'm not sure I get it but I'll try to answer. > > What I'm describing in the post is how logic can be expressed with generators > rather than callbacks. I'm assuming that the low level calls are > callback-style. So, there is no reference to any specific I/O library and/or > to back pressure. > > The back pressure problem is a problem that I'm handling in streamline's > streams module. And I'm handling it with a simple pair of async calls: > stream.read(cb) and stream.write(cb, buffer) that are small wrappers around > node streams. > > The read call pauses and resumes the underlying stream based on some > configurable high/low mark buffering limits (you can set them to 0 but then > the stream will pause every time it needs to buffer a chunk). > > The write call deals with the drain event under the hood. If the lower level > call write call returns true, the callback is called immediately (streamline > trampolines so there is no risk of stack overflow in callback mode). If it > returns false, the callback is triggered by the drain event. > > Pump loops can be written as: > > while (data = input.read(_)) > output.write(_, data); > > In callback mode, streamline transforms this loop into something like: > > (function loop() { > input.read(function(err, data) { > if (err) return cb(err); > if (data) > output.write(function(err) { > if (err) return cb(err); > loop(): > }, data); > else cb(); > }); > })(); > > In generators mode it transforms it into: > > while (data = yield input.read(_)) > yield output.write(_, data) > > The code looks very different but it execute just like the callback code > above. The input.read and output.write calls will use "invoke" and callbacks > to interact with the underlying node streams APIs. The run loop that I've > given in my post will exit at spot (c) every time a callback is pending and > the callback will reactivate it by calling resume; so, even though the code > does not look async and callback driven it is actually completely async and > callback driven. > > With this pump loop, backpressure happens naturally when the two calls are > combined together. If the output is slower than the input when pumping from > an input stream to an output stream, the write call will start to wait for > drain events. This will naturally stop the pump loop. The input stream will > continue to receive data events but it will just buffer because read won't be > called by the pump loop any more. When buffering goes over the high mark, the > input stream will be paused. Then, at some point, the output stream will > receive a drain event. It will call its callback, which will resume the pump > loop. Read will be called and will get data that has been buffered. The input > stream will be resumed when buffering goes below the low mark, etc., etc. If > the drain event comes before the input reaches the high water mark, the loop > will be resumed and the input stream won't be paused, which is what we want. > > So, even though the pump loop is written as a simple while (data = > input.read(_)) output.write(_, data), it does handle the back pressure. > > Bruno > > > On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote: > How do you handle back pressure? > > On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote: > >> Yes, I fixed it. Thanks. >> >> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote: >> On 5/19/2012 6:20 AM, Bruno Jouhier wrote: >> > http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/ >> > >> >> shouldn't that be print(num) not print(n) >> >> >> -- >> Job Board: http://jobs.nodejs.org/ >> Posting guidelines: >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> You received this message because you are subscribed to the Google >> Groups "nodejs" group. >> To post to this group, send email to [email protected] >> To unsubscribe from this group, send email to >> [email protected] >> For more options, visit this group at >> http://groups.google.com/group/nodejs?hl=en?hl=en > > > On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote: > How do you handle back pressure? > > On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote: > >> Yes, I fixed it. Thanks. >> >> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote: >> On 5/19/2012 6:20 AM, Bruno Jouhier wrote: >> > http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/ >> > >> >> shouldn't that be print(num) not print(n) >> >> >> -- >> Job Board: http://jobs.nodejs.org/ >> Posting guidelines: >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> You received this message because you are subscribed to the Google >> Groups "nodejs" group. >> To post to this group, send email to [email protected] >> To unsubscribe from this group, send email to >> [email protected] >> For more options, visit this group at >> http://groups.google.com/group/nodejs?hl=en?hl=en > > > On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote: > How do you handle back pressure? > > On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote: > >> Yes, I fixed it. Thanks. >> >> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote: >> On 5/19/2012 6:20 AM, Bruno Jouhier wrote: >> > http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/ >> > >> >> shouldn't that be print(num) not print(n) >> >> >> -- >> Job Board: http://jobs.nodejs.org/ >> Posting guidelines: >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> You received this message because you are subscribed to the Google >> Groups "nodejs" group. >> To post to this group, send email to [email protected] >> To unsubscribe from this group, send email to >> [email protected] >> For more options, visit this group at >> http://groups.google.com/group/nodejs?hl=en?hl=en > > > On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote: > How do you handle back pressure? > > On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote: > >> Yes, I fixed it. Thanks. >> >> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote: >> On 5/19/2012 6:20 AM, Bruno Jouhier wrote: >> > http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/ >> > >> >> shouldn't that be print(num) not print(n) >> >> >> -- >> Job Board: http://jobs.nodejs.org/ >> Posting guidelines: >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> You received this message because you are subscribed to the Google >> Groups "nodejs" group. >> To post to this group, send email to [email protected] >> To unsubscribe from this group, send email to >> [email protected] >> For more options, visit this group at >> http://groups.google.com/group/nodejs?hl=en?hl=en > > > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > > > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > > > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
