stream.end should uncork and +1 (or see my prev post) Am Mittwoch, 24. April 2013 15:18:08 UTC+2 schrieb Matt Sergeant: > > I'm gonna go against the crowd and ask for B:cork/uncork here. It's far > easier for when you don't have a bunch of pre-composed buffer objects, and > would fit into something streaming lines to the output a lot better. Yes > you can fuck up, but there's a million ways the programmer can fuck up with > node anyway. > > > On Wed, Apr 24, 2013 at 4:18 AM, Isaac Schlueter <[email protected]<javascript:> > > wrote: > >> > +1 C / D, D would be less awkward as far as building up the things >> > you're passing to .writev() goes, but the arrays are alright. Less >> > fancy stuff in core++ >> >> Part of my complaint about parallel arrays is that we'd probably end >> up having to re-match them in many cases anyway. An array of >> {chunk,encoding} is how the write queue is already implemented. >> >> >> On Tue, Apr 23, 2013 at 11:16 PM, Jake Verbaten >> <[email protected]<javascript:>> >> wrote: >> > I like D) stream.writev([ {chunk:buf, encoding: blerg}, ...], callback) >> > >> > The leveldb driver has a very similar batch api ( >> > https://github.com/rvagg/node-levelup#batch ) >> > >> > the leveldb driver also has a large thread about possible better APIs ( >> > https://github.com/rvagg/node-levelup/issues/45 ) from which some >> > inspiration may be drawn. >> > >> > >> > On Tue, Apr 23, 2013 at 9:27 PM, tjholowaychuk >> > <[email protected]<javascript:> >> > >> > wrote: >> >> >> >> +1 C / D, D would be less awkward as far as building up the things >> >> you're passing to .writev() goes, but the arrays are alright. Less >> >> fancy stuff in core++ >> >> >> >> On Apr 22, 5:01 pm, Isaac Schlueter <[email protected]> wrote: >> >> > There's a syscall called `writev` that lets you write an array (ie, >> >> > "Vector") of buffers of data rather than a single buffer. >> >> > >> >> > I'd like to support something like this for Streams in Node, mostly >> >> > because it will allow us to save a lot of TCP write() calls, without >> >> > having to copy data around, especially for chunked encoding writes. >> >> > (We write a lot of tiny buffers for HTTP, it's kind of a nightmare, >> >> > actually.) >> >> > >> >> > Fedor Indutny has already done basically all of the legwork to >> >> > implement this. Where we're stuck is the API surface, and here are >> >> > some options. Node is not a democracy, but your vote counts anyway, >> >> > especially if it's a really good vote with some really good argument >> >> > behind it :) >> >> > >> >> > Goals: >> >> > 1. Make http more good. >> >> > 2. Don't break existing streams. >> >> > 3. Don't make things hard. >> >> > 4. Don't be un-node-ish >> >> > >> >> > For all of these, batched writes will only be available if the >> >> > Writable stream implements a `_writev()` method. No _writev, no >> >> > batched writes. Any bulk writes will just be passed to _write(chunk, >> >> > encoding, callback) one at a time in the order received. >> >> > >> >> > In all cases, any queued writes will be passed to _writev if that >> >> > function is implemented, even if they're just backed up from a slow >> >> > connection. >> >> > >> >> > Ideas: >> >> > >> >> > A) stream.bulk(function() { stream.write('hello'); >> >> > stream.write('world'); stream.end('!\n') }) >> >> > >> >> > Any writes done in the function passed to `stream.bulk()` will be >> >> > batched into a single writev. >> >> > >> >> > Upside: >> >> > - Easier to not fuck up and stay frozen forever. There is basically >> >> > zero chance that you'll leave the stream in a corked state. (Same >> >> > reason why domain.run() is better than enter()/exit().) >> >> > >> >> > Downsides: >> >> > - easier to fuck up and not actually batch things. eg, >> >> > s.bulk(function(){setTimeout(...)}) >> >> > - bulk is a weird name. "batch" maybe? Nothing else really seems >> >> > appropriate either. >> >> > - somewhat inflexible, since all writes have to be done in the same >> >> > function call >> >> > >> >> > B) stream.cork(); stream.write('hello'); stream.write('world'); >> >> > stream.end('!\n'); stream.uncork(); >> >> > >> >> > Any writes done while corked will be flushed to _writev() when >> uncorked. >> >> > >> >> > Upside: >> >> > - Easy to implement >> >> > - Strictly more flexible than stream.bulk(writer). (Can trivially >> >> > implement a bulk function using cork/uncork) >> >> > - Useful for cases outside of writev (like corking a http request >> >> > until the connection is established) >> >> > >> >> > Downsides: >> >> > - Easy to fuck up and stay corked forever. >> >> > - Two functions instead of just one (double the surface area >> increase) >> >> > >> >> > C) stream.writev([chunks,...], [encodings,...], callback) >> >> > >> >> > That is, implement a first-class top-level function called writev() >> >> > which you can call with an array of chunks and an array of encodings. >> >> > >> >> > Upside: >> >> > - No unnecessary surface area increase >> >> > - NOW IT'S YOUR PROBLEM, NOT MINE, HAHA! (Seriously, though, it's >> >> > less magical, simpler stream.Writable implementation, etc.) >> >> > >> >> > Downside: >> >> > - A little bit tricky when you don't already have a list of chunks to >> >> > send. (For example, with cork, you could write a bunch of stuff into >> >> > it, and then uncork all at the end, and do one writev, even if it >> took >> >> > a few ms to get it all.) >> >> > - parallel arrays, ew. >> >> > >> >> > D) stream.writev([ {chunk:buf, encoding: blerg}, ...], callback) >> >> > >> >> > That is, same as C, but with an array of {chunk,encoding} objects >> >> > instead of the parallel arrays. >> >> > >> >> > Same +/- as C, except the parallel array bit. This is probably how >> >> > we'd call the implementation's stream._writev() anyway, so it'd be a >> >> > bit simpler. >> >> > >> >> > Which of these seems like it makes the most sense to you? >> >> > >> >> > Is there another approach that you'd like to see here? (Note: "save >> >> > all writes until end of tick always" and "copy into one big buffer" >> >> > approaches are not feasible for obvious performance reasons.) >> >> >> >> -- >> >> -- >> >> Job Board: http://jobs.nodejs.org/ >> >> Posting guidelines: >> >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> >> You received this message because you are subscribed to the Google >> >> Groups "nodejs" group. >> >> To post to this group, send email to [email protected]<javascript:> >> >> To unsubscribe from this group, send email to >> >> [email protected] <javascript:> >> >> For more options, visit this group at >> >> http://groups.google.com/group/nodejs?hl=en?hl=en >> >> >> >> --- >> >> You received this message because you are subscribed to the Google >> Groups >> >> "nodejs" group. >> >> To unsubscribe from this group and stop receiving emails from it, send >> an >> >> email to [email protected] <javascript:>. >> >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> >> >> > >> > -- >> > -- >> > Job Board: http://jobs.nodejs.org/ >> > Posting guidelines: >> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> > You received this message because you are subscribed to the Google >> > Groups "nodejs" group. >> > To post to this group, send email to [email protected]<javascript:> >> > To unsubscribe from this group, send email to >> > [email protected] <javascript:> >> > For more options, visit this group at >> > http://groups.google.com/group/nodejs?hl=en?hl=en >> > >> > --- >> > You received this message because you are subscribed to the Google >> Groups >> > "nodejs" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an >> > email to [email protected] <javascript:>. >> > For more options, visit https://groups.google.com/groups/opt_out. >> > >> > >> >> -- >> -- >> Job Board: http://jobs.nodejs.org/ >> Posting guidelines: >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> You received this message because you are subscribed to the Google >> Groups "nodejs" group. >> To post to this group, send email to [email protected]<javascript:> >> To unsubscribe from this group, send email to >> [email protected] <javascript:> >> For more options, visit this group at >> http://groups.google.com/group/nodejs?hl=en?hl=en >> >> --- >> You received this message because you are subscribed to the Google Groups >> "nodejs" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> >
-- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
