On Mon, May 21, 2012 at 3:16 PM, Mikeal Rogers <[email protected]>wrote:
> > On May 21, 2012, at May 21, 20121:13 PM, Tim Caswell wrote: > > > > On Mon, May 21, 2012 at 3:08 PM, Mikeal Rogers <[email protected]>wrote: > >> This has *nothing* to do with callbacks. Nothing whatsoever. >> >> You're structuring use cases around behavior that we don't actually use >> callbacks for in node.js, we use streams. Anything that hooks in to a file >> descriptor uses a stream. >> > > Would it help if Bruno used examples with real business logic instead of > teaching the mechanics of streamline using code patterns we all understand? > I'm pretty sure you can still use .pipe() in streamline code too. > > >> >> All of these modes; callbacks, fibers, and generators don't handle back >> pressure. Callbacks in node.js don't handle back pressure, which is why we >> have streams. >> > > Callbacks can handle backpressure. I've done it in a system where I can't > use prototypes or return values. It's just not how node does it natively. > > > They can if you write it all yourself, inline, the way we had to in node > 0.1.60 or whatever. But callbacks push the burden of this solution all the > way to the top of the application logic which is not recommended. > > With streams we have a good abstraction, that others can build on, which > solves this problem well, without indentation, in a single line. > Ok, so if generators and callbacks have nothing to do with streams, then why did you bring up streams in this thread? I don't understand. Was Bruno proposing that we stop using node's streams and go back to using callbacks for everything? > > > > >> >> On May 21, 2012, at May 21, 201212:58 PM, Bruno Jouhier wrote: >> >> One thing I'm not getting is what coros have to do in all this. I can use >> streamline in 3 modes (callbacks, fibers and generators) which are >> equivalent (the streamline source behaves the same, even though it gets >> compiled differently). >> The last 2 modes use some form of coroutines but the first one does not. >> So when I use the streamline streams module in callbacks mode, there is no >> coro involved at all, it is pure JS with callbacks. >> >> On Monday, May 21, 2012 9:18:07 PM UTC+2, Mikeal Rogers wrote: >>> >>> >>> On May 21, 2012, at May 21, 201212:03 PM, Bruno Jouhier wrote: >>> >>> luvmonkey really helped me but I got a bit bored playing with the timer >>> towards the end. Will be nice to have more APIs >>> >>> Regarding the backpressure issue I forgot to mention that there is no >>> explicit backpressure handling logic in the pump loop itself (it is just a >>> loop). It works sorta "naturally" thanks to the events dispatched by the >>> event loop. This means that it will work with arbitrary topologies (for >>> example several inputs being joined into one output, one input being >>> dispatched to several outputs, etc). It also works with arbitrary logic in >>> the middle (complex transforms, etc.). So it is a decoupled and flexible >>> solution. >>> >>> >>> In theory, but not in practice. >>> >>> As we've already seen with streams you can't generalize the back >>> pressure logic for multi stream outputs. Everyone who is actually doing >>> this in production has their own application specific versions of pipe() >>> that decide when and if the inputs should actually be paused. >>> >>> I'm skeptical you're actually handling the back pressure case properly. >>> coro systems in Ruby and Python don't really handle it either and it's a >>> problem with generators that we've spent a lot of time discussing and >>> exploring. This debate, in javascript, goes all the way back to the JSGI >>> discussions before node had streams and the eJSGI work that @isaacs did >>> which also pre-dates streams. >>> >>> Back pressure from file descriptors, sockets in particular, is one of >>> the most important parts of node streams and is crucial to concurrent >>> performance with mobile client. If you're not prioritizing it then I can't >>> take this project seriously. If you are then I'd be very interested to see >>> how you tackle the issues we've explored already in the work that pre-dated >>> streams. >>> >>> Expressing logic as generators rather than callbacks is a decent >>> academic experiment but it's not tackling the hard problems we tackle with >>> streams, nor are the comparisons as compelling since streams being piped >>> together doesn't expose any callback indentation. Ignoring these cases or >>> contriving examples with file descriptors that aren't using streams is a >>> very thin strawman. >>> >>> >>> >>> On Monday, May 21, 2012 4:20:56 PM UTC+2, Tim Caswell wrote: >>>> >>>> Nice work! Now I've got more motivation to get LuvMonkey into a more >>>> usable state. >>>> >>>> On Sat, May 19, 2012 at 4:06 PM, Bruno Jouhier <[email protected]>wrote: >>>> >>>>> I'm not sure I get it but I'll try to answer. >>>>> >>>>> What I'm describing in the post is how logic can be expressed with >>>>> generators rather than callbacks. I'm assuming that the low level calls >>>>> are >>>>> callback-style. So, there is no reference to any specific I/O library >>>>> and/or to back pressure. >>>>> >>>>> The back pressure problem is a problem that I'm handling in >>>>> streamline's streams module. And I'm handling it with a simple pair of >>>>> async calls: stream.read(cb) and stream.write(cb, buffer) that are small >>>>> wrappers around node streams. >>>>> >>>>> The read call pauses and resumes the underlying stream based on some >>>>> configurable high/low mark buffering limits (you can set them to 0 but >>>>> then >>>>> the stream will pause every time it needs to buffer a chunk). >>>>> >>>>> The write call deals with the drain event under the hood. If the lower >>>>> level call write call returns true, the callback is called immediately >>>>> (streamline trampolines so there is no risk of stack overflow in callback >>>>> mode). If it returns false, the callback is triggered by the drain event. >>>>> >>>>> Pump loops can be written as: >>>>> >>>>> while (data = input.read(_)) >>>>> output.write(_, data); >>>>> >>>>> In callback mode, streamline transforms this loop into something like: >>>>> >>>>> (function loop() { >>>>> input.read(function(err, data) { >>>>> if (err) return cb(err); >>>>> if (data) >>>>> output.write(function(err) { >>>>> if (err) return cb(err); >>>>> loop(): >>>>> }, data); >>>>> else cb(); >>>>> }); >>>>> })(); >>>>> >>>>> In generators mode it transforms it into: >>>>> >>>>> while (data = yield input.read(_)) >>>>> yield output.write(_, data) >>>>> >>>>> The code looks very different but it execute just like the callback >>>>> code above. The input.read and output.write calls will use "invoke" and >>>>> callbacks to interact with the underlying node streams APIs. The run loop >>>>> that I've given in my post will exit at spot (c) every time a callback is >>>>> pending and the callback will reactivate it by calling resume; so, even >>>>> though the code does not look async and callback driven it is actually >>>>> completely async and callback driven. >>>>> >>>>> With this pump loop, backpressure happens naturally when the two calls >>>>> are combined together. If the output is slower than the input when pumping >>>>> from an input stream to an output stream, the write call will start to >>>>> wait >>>>> for drain events. This will naturally stop the pump loop. The input stream >>>>> will continue to receive data events but it will just buffer because read >>>>> won't be called by the pump loop any more. When buffering goes over the >>>>> high mark, the input stream will be paused. Then, at some point, the >>>>> output >>>>> stream will receive a drain event. It will call its callback, which will >>>>> resume the pump loop. Read will be called and will get data that has been >>>>> buffered. The input stream will be resumed when buffering goes below the >>>>> low mark, etc., etc. If the drain event comes before the input reaches the >>>>> high water mark, the loop will be resumed and the input stream won't be >>>>> paused, which is what we want. >>>>> >>>>> So, even though the pump loop is written as a simple while (data = >>>>> input.read(_)) output.write(_, data), it does handle the back pressure. >>>>> >>>>> Bruno >>>>> >>>>> >>>>> On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote: >>>>>> >>>>>> How do you handle back pressure? >>>>>> >>>>>> On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote: >>>>>> >>>>>> Yes, I fixed it. Thanks. >>>>>> >>>>>> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote: >>>>>>> >>>>>>> On 5/19/2012 6:20 AM, Bruno Jouhier wrote: >>>>>>> > http://bjouhier.wordpress.com/****2012/05/18/asynchronous-**javasc >>>>>>> **ript-with-generators-an-**experi**ment/<http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/> >>>>>>> >>>>>>> shouldn't that be print(num) not print(n) >>>>>>> >>>>>>> >>>>>> -- >>>>>> Job Board: http://jobs.nodejs.org/ >>>>>> Posting guidelines: https://github.com/joyent/**node** >>>>>> /wiki/Mailing-List-**Posting-**Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "nodejs" group. >>>>>> To post to this group, send email to [email protected] >>>>>> To unsubscribe from this group, send email to >>>>>> nodejs+unsubscribe@**googlegroup**s.com<[email protected]> >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/**group**/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>>>>> >>>>>> >>>>>> >>>>> On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote: >>>>>> >>>>>> How do you handle back pressure? >>>>>> >>>>>> On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote: >>>>>> >>>>>> Yes, I fixed it. Thanks. >>>>>> >>>>>> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote: >>>>>>> >>>>>>> On 5/19/2012 6:20 AM, Bruno Jouhier wrote: >>>>>>> > http://bjouhier.wordpress.com/****2012/05/18/asynchronous-**javasc >>>>>>> **ript-with-generators-an-**experi**ment/<http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/> >>>>>>> >>>>>>> shouldn't that be print(num) not print(n) >>>>>>> >>>>>>> >>>>>> -- >>>>>> Job Board: http://jobs.nodejs.org/ >>>>>> Posting guidelines: https://github.com/joyent/**node** >>>>>> /wiki/Mailing-List-**Posting-**Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "nodejs" group. >>>>>> To post to this group, send email to [email protected] >>>>>> To unsubscribe from this group, send email to >>>>>> nodejs+unsubscribe@**googlegroup**s.com<[email protected]> >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/**group**/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>>>>> >>>>>> >>>>>> >>>>> On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote: >>>>>> >>>>>> How do you handle back pressure? >>>>>> >>>>>> On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote: >>>>>> >>>>>> Yes, I fixed it. Thanks. >>>>>> >>>>>> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote: >>>>>>> >>>>>>> On 5/19/2012 6:20 AM, Bruno Jouhier wrote: >>>>>>> > http://bjouhier.wordpress.com/****2012/05/18/asynchronous-**javasc >>>>>>> **ript-with-generators-an-**experi**ment/<http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/> >>>>>>> >>>>>>> shouldn't that be print(num) not print(n) >>>>>>> >>>>>>> >>>>>> -- >>>>>> Job Board: http://jobs.nodejs.org/ >>>>>> Posting guidelines: https://github.com/joyent/**node** >>>>>> /wiki/Mailing-List-**Posting-**Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "nodejs" group. >>>>>> To post to this group, send email to [email protected] >>>>>> To unsubscribe from this group, send email to >>>>>> nodejs+unsubscribe@**googlegroup**s.com<[email protected]> >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/**group**/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>>>>> >>>>>> >>>>>> >>>>> On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote: >>>>>> >>>>>> How do you handle back pressure? >>>>>> >>>>>> On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote: >>>>>> >>>>>> Yes, I fixed it. Thanks. >>>>>> >>>>>> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote: >>>>>>> >>>>>>> On 5/19/2012 6:20 AM, Bruno Jouhier wrote: >>>>>>> > http://bjouhier.wordpress.com/****2012/05/18/asynchronous-**javasc >>>>>>> **ript-with-generators-an-**experi**ment/<http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/> >>>>>>> >>>>>>> shouldn't that be print(num) not print(n) >>>>>>> >>>>>>> >>>>>> -- >>>>>> Job Board: http://jobs.nodejs.org/ >>>>>> Posting guidelines: https://github.com/joyent/**node** >>>>>> /wiki/Mailing-List-**Posting-**Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "nodejs" group. >>>>>> To post to this group, send email to [email protected] >>>>>> To unsubscribe from this group, send email to >>>>>> nodejs+unsubscribe@**googlegroup**s.com<[email protected]> >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/**group**/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>>>>> >>>>>> >>>>>> >>>>> -- >>>>> Job Board: http://jobs.nodejs.org/ >>>>> Posting guidelines: https://github.com/joyent/** >>>>> node/wiki/Mailing-List-**Posting-Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>>>> You received this message because you are subscribed to the Google >>>>> Groups "nodejs" group. >>>>> To post to this group, send email to [email protected] >>>>> To unsubscribe from this group, send email to >>>>> nodejs+unsubscribe@**googlegroups.com<nodejs%[email protected]> >>>>> For more options, visit this group at >>>>> http://groups.google.com/**group/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>>>> >>>> >>>> >>> -- >>> Job Board: http://jobs.nodejs.org/ >>> Posting guidelines: https://github.com/joyent/**node/wiki/Mailing-List-* >>> *Posting-Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>> You received this message because you are subscribed to the Google >>> Groups "nodejs" group. >>> To post to this group, send email to [email protected] >>> To unsubscribe from this group, send email to >>> nodejs+unsubscribe@**googlegroups.com<[email protected]> >>> For more options, visit this group at >>> http://groups.google.com/**group/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>> >>> >>> >> -- >> Job Board: http://jobs.nodejs.org/ >> Posting guidelines: >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> You received this message because you are subscribed to the Google >> Groups "nodejs" group. >> To post to this group, send email to [email protected] >> To unsubscribe from this group, send email to >> [email protected] >> For more options, visit this group at >> http://groups.google.com/group/nodejs?hl=en?hl=en >> >> >> >> -- >> Job Board: http://jobs.nodejs.org/ >> Posting guidelines: >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> You received this message because you are subscribed to the Google >> Groups "nodejs" group. >> To post to this group, send email to [email protected] >> To unsubscribe from this group, send email to >> [email protected] >> For more options, visit this group at >> http://groups.google.com/group/nodejs?hl=en?hl=en >> > > > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > > > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
