This has *nothing* to do with callbacks. Nothing whatsoever. You're structuring use cases around behavior that we don't actually use callbacks for in node.js, we use streams. Anything that hooks in to a file descriptor uses a stream.
All of these modes; callbacks, fibers, and generators don't handle back pressure. Callbacks in node.js don't handle back pressure, which is why we have streams. On May 21, 2012, at May 21, 201212:58 PM, Bruno Jouhier wrote: > One thing I'm not getting is what coros have to do in all this. I can use > streamline in 3 modes (callbacks, fibers and generators) which are equivalent > (the streamline source behaves the same, even though it gets compiled > differently). > The last 2 modes use some form of coroutines but the first one does not. So > when I use the streamline streams module in callbacks mode, there is no coro > involved at all, it is pure JS with callbacks. > > On Monday, May 21, 2012 9:18:07 PM UTC+2, Mikeal Rogers wrote: > > On May 21, 2012, at May 21, 201212:03 PM, Bruno Jouhier wrote: > >> luvmonkey really helped me but I got a bit bored playing with the timer >> towards the end. Will be nice to have more APIs >> >> Regarding the backpressure issue I forgot to mention that there is no >> explicit backpressure handling logic in the pump loop itself (it is just a >> loop). It works sorta "naturally" thanks to the events dispatched by the >> event loop. This means that it will work with arbitrary topologies (for >> example several inputs being joined into one output, one input being >> dispatched to several outputs, etc). It also works with arbitrary logic in >> the middle (complex transforms, etc.). So it is a decoupled and flexible >> solution. > > In theory, but not in practice. > > As we've already seen with streams you can't generalize the back pressure > logic for multi stream outputs. Everyone who is actually doing this in > production has their own application specific versions of pipe() that decide > when and if the inputs should actually be paused. > > I'm skeptical you're actually handling the back pressure case properly. coro > systems in Ruby and Python don't really handle it either and it's a problem > with generators that we've spent a lot of time discussing and exploring. This > debate, in javascript, goes all the way back to the JSGI discussions before > node had streams and the eJSGI work that @isaacs did which also pre-dates > streams. > > Back pressure from file descriptors, sockets in particular, is one of the > most important parts of node streams and is crucial to concurrent performance > with mobile client. If you're not prioritizing it then I can't take this > project seriously. If you are then I'd be very interested to see how you > tackle the issues we've explored already in the work that pre-dated streams. > > Expressing logic as generators rather than callbacks is a decent academic > experiment but it's not tackling the hard problems we tackle with streams, > nor are the comparisons as compelling since streams being piped together > doesn't expose any callback indentation. Ignoring these cases or contriving > examples with file descriptors that aren't using streams is a very thin > strawman. > >> >> >> On Monday, May 21, 2012 4:20:56 PM UTC+2, Tim Caswell wrote: >> Nice work! Now I've got more motivation to get LuvMonkey into a more usable >> state. >> >> On Sat, May 19, 2012 at 4:06 PM, Bruno Jouhier <[email protected]> wrote: >> I'm not sure I get it but I'll try to answer. >> >> What I'm describing in the post is how logic can be expressed with >> generators rather than callbacks. I'm assuming that the low level calls are >> callback-style. So, there is no reference to any specific I/O library and/or >> to back pressure. >> >> The back pressure problem is a problem that I'm handling in streamline's >> streams module. And I'm handling it with a simple pair of async calls: >> stream.read(cb) and stream.write(cb, buffer) that are small wrappers around >> node streams. >> >> The read call pauses and resumes the underlying stream based on some >> configurable high/low mark buffering limits (you can set them to 0 but then >> the stream will pause every time it needs to buffer a chunk). >> >> The write call deals with the drain event under the hood. If the lower level >> call write call returns true, the callback is called immediately (streamline >> trampolines so there is no risk of stack overflow in callback mode). If it >> returns false, the callback is triggered by the drain event. >> >> Pump loops can be written as: >> >> while (data = input.read(_)) >> output.write(_, data); >> >> In callback mode, streamline transforms this loop into something like: >> >> (function loop() { >> input.read(function(err, data) { >> if (err) return cb(err); >> if (data) >> output.write(function(err) { >> if (err) return cb(err); >> loop(): >> }, data); >> else cb(); >> }); >> })(); >> >> In generators mode it transforms it into: >> >> while (data = yield input.read(_)) >> yield output.write(_, data) >> >> The code looks very different but it execute just like the callback code >> above. The input.read and output.write calls will use "invoke" and callbacks >> to interact with the underlying node streams APIs. The run loop that I've >> given in my post will exit at spot (c) every time a callback is pending and >> the callback will reactivate it by calling resume; so, even though the code >> does not look async and callback driven it is actually completely async and >> callback driven. >> >> With this pump loop, backpressure happens naturally when the two calls are >> combined together. If the output is slower than the input when pumping from >> an input stream to an output stream, the write call will start to wait for >> drain events. This will naturally stop the pump loop. The input stream will >> continue to receive data events but it will just buffer because read won't >> be called by the pump loop any more. When buffering goes over the high mark, >> the input stream will be paused. Then, at some point, the output stream will >> receive a drain event. It will call its callback, which will resume the pump >> loop. Read will be called and will get data that has been buffered. The >> input stream will be resumed when buffering goes below the low mark, etc., >> etc. If the drain event comes before the input reaches the high water mark, >> the loop will be resumed and the input stream won't be paused, which is what >> we want. >> >> So, even though the pump loop is written as a simple while (data = >> input.read(_)) output.write(_, data), it does handle the back pressure. >> >> Bruno >> >> >> On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote: >> How do you handle back pressure? >> >> On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote: >> >>> Yes, I fixed it. Thanks. >>> >>> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote: >>> On 5/19/2012 6:20 AM, Bruno Jouhier wrote: >>> > http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/ >>> > >>> >>> shouldn't that be print(num) not print(n) >>> >>> >>> -- >>> Job Board: http://jobs.nodejs.org/ >>> Posting guidelines: >>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >>> You received this message because you are subscribed to the Google >>> Groups "nodejs" group. >>> To post to this group, send email to [email protected] >>> To unsubscribe from this group, send email to >>> [email protected] >>> For more options, visit this group at >>> http://groups.google.com/group/nodejs?hl=en?hl=en >> >> >> On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote: >> How do you handle back pressure? >> >> On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote: >> >>> Yes, I fixed it. Thanks. >>> >>> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote: >>> On 5/19/2012 6:20 AM, Bruno Jouhier wrote: >>> > http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/ >>> > >>> >>> shouldn't that be print(num) not print(n) >>> >>> >>> -- >>> Job Board: http://jobs.nodejs.org/ >>> Posting guidelines: >>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >>> You received this message because you are subscribed to the Google >>> Groups "nodejs" group. >>> To post to this group, send email to [email protected] >>> To unsubscribe from this group, send email to >>> [email protected] >>> For more options, visit this group at >>> http://groups.google.com/group/nodejs?hl=en?hl=en >> >> >> On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote: >> How do you handle back pressure? >> >> On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote: >> >>> Yes, I fixed it. Thanks. >>> >>> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote: >>> On 5/19/2012 6:20 AM, Bruno Jouhier wrote: >>> > http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/ >>> > >>> >>> shouldn't that be print(num) not print(n) >>> >>> >>> -- >>> Job Board: http://jobs.nodejs.org/ >>> Posting guidelines: >>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >>> You received this message because you are subscribed to the Google >>> Groups "nodejs" group. >>> To post to this group, send email to [email protected] >>> To unsubscribe from this group, send email to >>> [email protected] >>> For more options, visit this group at >>> http://groups.google.com/group/nodejs?hl=en?hl=en >> >> >> On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote: >> How do you handle back pressure? >> >> On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote: >> >>> Yes, I fixed it. Thanks. >>> >>> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote: >>> On 5/19/2012 6:20 AM, Bruno Jouhier wrote: >>> > http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/ >>> > >>> >>> shouldn't that be print(num) not print(n) >>> >>> >>> -- >>> Job Board: http://jobs.nodejs.org/ >>> Posting guidelines: >>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >>> You received this message because you are subscribed to the Google >>> Groups "nodejs" group. >>> To post to this group, send email to [email protected] >>> To unsubscribe from this group, send email to >>> [email protected] >>> For more options, visit this group at >>> http://groups.google.com/group/nodejs?hl=en?hl=en >> >> >> -- >> Job Board: http://jobs.nodejs.org/ >> Posting guidelines: >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> You received this message because you are subscribed to the Google >> Groups "nodejs" group. >> To post to this group, send email to [email protected] >> To unsubscribe from this group, send email to >> [email protected] >> For more options, visit this group at >> http://groups.google.com/group/nodejs?hl=en?hl=en >> >> >> -- >> Job Board: http://jobs.nodejs.org/ >> Posting guidelines: >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> You received this message because you are subscribed to the Google >> Groups "nodejs" group. >> To post to this group, send email to [email protected] >> To unsubscribe from this group, send email to >> [email protected] >> For more options, visit this group at >> http://groups.google.com/group/nodejs?hl=en?hl=en > > > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
