> +1 C / D, D would be less awkward as far as building up the things
> you're passing to .writev() goes, but the arrays are alright. Less
> fancy stuff in core++
Part of my complaint about parallel arrays is that we'd probably end
up having to re-match them in many cases anyway. An array of
{chunk,encoding} is how the write queue is already implemented.
On Tue, Apr 23, 2013 at 11:16 PM, Jake Verbaten <[email protected]> wrote:
> I like D) stream.writev([ {chunk:buf, encoding: blerg}, ...], callback)
>
> The leveldb driver has a very similar batch api (
> https://github.com/rvagg/node-levelup#batch )
>
> the leveldb driver also has a large thread about possible better APIs (
> https://github.com/rvagg/node-levelup/issues/45 ) from which some
> inspiration may be drawn.
>
>
> On Tue, Apr 23, 2013 at 9:27 PM, tjholowaychuk <[email protected]>
> wrote:
>>
>> +1 C / D, D would be less awkward as far as building up the things
>> you're passing to .writev() goes, but the arrays are alright. Less
>> fancy stuff in core++
>>
>> On Apr 22, 5:01 pm, Isaac Schlueter <[email protected]> wrote:
>> > There's a syscall called `writev` that lets you write an array (ie,
>> > "Vector") of buffers of data rather than a single buffer.
>> >
>> > I'd like to support something like this for Streams in Node, mostly
>> > because it will allow us to save a lot of TCP write() calls, without
>> > having to copy data around, especially for chunked encoding writes.
>> > (We write a lot of tiny buffers for HTTP, it's kind of a nightmare,
>> > actually.)
>> >
>> > Fedor Indutny has already done basically all of the legwork to
>> > implement this. Where we're stuck is the API surface, and here are
>> > some options. Node is not a democracy, but your vote counts anyway,
>> > especially if it's a really good vote with some really good argument
>> > behind it :)
>> >
>> > Goals:
>> > 1. Make http more good.
>> > 2. Don't break existing streams.
>> > 3. Don't make things hard.
>> > 4. Don't be un-node-ish
>> >
>> > For all of these, batched writes will only be available if the
>> > Writable stream implements a `_writev()` method. No _writev, no
>> > batched writes. Any bulk writes will just be passed to _write(chunk,
>> > encoding, callback) one at a time in the order received.
>> >
>> > In all cases, any queued writes will be passed to _writev if that
>> > function is implemented, even if they're just backed up from a slow
>> > connection.
>> >
>> > Ideas:
>> >
>> > A) stream.bulk(function() { stream.write('hello');
>> > stream.write('world'); stream.end('!\n') })
>> >
>> > Any writes done in the function passed to `stream.bulk()` will be
>> > batched into a single writev.
>> >
>> > Upside:
>> > - Easier to not fuck up and stay frozen forever. There is basically
>> > zero chance that you'll leave the stream in a corked state. (Same
>> > reason why domain.run() is better than enter()/exit().)
>> >
>> > Downsides:
>> > - easier to fuck up and not actually batch things. eg,
>> > s.bulk(function(){setTimeout(...)})
>> > - bulk is a weird name. "batch" maybe? Nothing else really seems
>> > appropriate either.
>> > - somewhat inflexible, since all writes have to be done in the same
>> > function call
>> >
>> > B) stream.cork(); stream.write('hello'); stream.write('world');
>> > stream.end('!\n'); stream.uncork();
>> >
>> > Any writes done while corked will be flushed to _writev() when uncorked.
>> >
>> > Upside:
>> > - Easy to implement
>> > - Strictly more flexible than stream.bulk(writer). (Can trivially
>> > implement a bulk function using cork/uncork)
>> > - Useful for cases outside of writev (like corking a http request
>> > until the connection is established)
>> >
>> > Downsides:
>> > - Easy to fuck up and stay corked forever.
>> > - Two functions instead of just one (double the surface area increase)
>> >
>> > C) stream.writev([chunks,...], [encodings,...], callback)
>> >
>> > That is, implement a first-class top-level function called writev()
>> > which you can call with an array of chunks and an array of encodings.
>> >
>> > Upside:
>> > - No unnecessary surface area increase
>> > - NOW IT'S YOUR PROBLEM, NOT MINE, HAHA! (Seriously, though, it's
>> > less magical, simpler stream.Writable implementation, etc.)
>> >
>> > Downside:
>> > - A little bit tricky when you don't already have a list of chunks to
>> > send. (For example, with cork, you could write a bunch of stuff into
>> > it, and then uncork all at the end, and do one writev, even if it took
>> > a few ms to get it all.)
>> > - parallel arrays, ew.
>> >
>> > D) stream.writev([ {chunk:buf, encoding: blerg}, ...], callback)
>> >
>> > That is, same as C, but with an array of {chunk,encoding} objects
>> > instead of the parallel arrays.
>> >
>> > Same +/- as C, except the parallel array bit. This is probably how
>> > we'd call the implementation's stream._writev() anyway, so it'd be a
>> > bit simpler.
>> >
>> > Which of these seems like it makes the most sense to you?
>> >
>> > Is there another approach that you'd like to see here? (Note: "save
>> > all writes until end of tick always" and "copy into one big buffer"
>> > approaches are not feasible for obvious performance reasons.)
>>
>> --
>> --
>> Job Board: http://jobs.nodejs.org/
>> Posting guidelines:
>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>> You received this message because you are subscribed to the Google
>> Groups "nodejs" group.
>> To post to this group, send email to [email protected]
>> To unsubscribe from this group, send email to
>> [email protected]
>> For more options, visit this group at
>> http://groups.google.com/group/nodejs?hl=en?hl=en
>>
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "nodejs" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>
> --
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to [email protected]
> To unsubscribe from this group, send email to
> [email protected]
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups
> "nodejs" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups
"nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
