I can see what you're saying, but the node docs do say<http://nodejs.org/api/stream.html#stream_readable_read_size_1>that if you don't pass a size argument to stream.read then the entire contents of the internal buffer are returned.
In any case, this would all be a lot easier if the readable event were guaranteed to fire when a new readable listener is registered for the first time. -- Michael Jackson @mjackson On Mon, Mar 25, 2013 at 3:20 PM, Dan Milon <[email protected]> wrote: > That's not guaranteed to work. > > You're assuming that `stream.read()` will return the whole internal > buffer, which is not documented anywhere. > > The right approach is to call `.read()` until it returns null. > Something like that: > > function collectStream(stream, cb) { > var bufs = [] > > function read() { > var chunk > > while ((chunk = stream.read()) != null) { > bufs.push(chunk) > } > } > > stream.on('error', cb) > > stream.on('readable', read) > > stream.on('end', function () { > cb(null, Buffer.concat(bufs)) > }) > > read() > } > > On 03/25/13 23:55, Michael Jackson wrote: > > Ok, that makes sense. > > > > So the readable event is more of an advisory event. The docs > > should probably say something about how you could possibly miss the > > event entirely if you're doing some other IO before you try and > > read from the stream. > > > > For posterity's sake, I adjusted my previous example: > > > > var http = require('http'); > > > > http.get('http://www.google.com', function (response) { > > console.log('got response with status ' + response.statusCode); > > > > setTimeout(function () { bufferStream(response, function (err, > > buffer) { console.log(buffer.toString()); }); }, 1000); }); > > > > function bufferStream(stream, callback) { var chunks = []; > > > > var chunk = stream.read(); if (chunk) { chunks.push(chunk); } > > > > stream.on('readable', function () { chunks.push(stream.read()); > > }); > > > > stream.on('error', function (error) { callback(error); }); > > > > stream.on('end', function () { callback(null, > > Buffer.concat(chunks)); }); } > > > > You can use the bufferStream function to catch all data on the > > stream, no matter how far in the future you are. > > > > -- Michael Jackson @mjackson > > > > > > On Mon, Mar 25, 2013 at 1:49 PM, Dean Landolt > > <[email protected] <mailto:[email protected]>> wrote: > > > > You can always call `stream.read`, at any time. This is how data > > is /pulled/ off the stream (instead of it being pushed to you, > > whether you're ready or not). Because of this you won't lose any > > data. With new streams there's no real notion of a paused state -- > > it's always paused. Once you grok that it may not seem so > > counter-intuitive. > > > > The `readable` event is like a corollary to `drain` -- there to > > tell you that it's worth bothering with a call to read. You don't > > /have/ to listen for it -- a (needlessly inefficient) stream reader > > could just as easily poll stream.read for new data periodically. > > > > > > On Mon, Mar 25, 2013 at 4:42 PM, Michael Jackson > > <[email protected] <mailto:[email protected]>> wrote: > > > > readable is emitted after you've actually started reading. > > > > > > That's not what it says in the docs > > <http://nodejs.org/api/stream.html#stream_event_readable>. > > > > ### Event: 'readable' When there is data ready to be consumed, this > > event will fire. When this event emits, call the read() method to > > consume the data. ### > > > > Calling stream.read *before* you get the "readable" event is > > totally counterintuitive. > > > > -- Michael Jackson @mjackson > > > > In your example, you dont ever `response.read()`, so no readable > > event is ever emitted. > > > > As you said, streams start in paused state and ready to be read. > > > > On 03/25/13 22:28, Michael Jackson wrote: > >> Is it correct to assume that a Readable won't emit the > > "readable" event > >> until you're registered for it? > >> > >> Reading through the streams2 docs, I was under the > > impression that all > >> streams start out paused and don't start emitting data > > until you add > >> either a "data" (for old streams) or a "readable" > > listener. For new > >> streams, this should mean that they don't emit "readable" > > until at least > >> one listener is registered. Otherwise we still need to do > > some buffering > >> in order to capture all the data. > >> > >> For example, this code misses the readable event on node 0.10: > >> > >> var http = require('http'); > >> > >> http.get('http://www.google.com', function (response) { > >> console.log('got response with status ' + > > response.statusCode); > >> > >> setTimeout(function () { response.on('readable', function () { > >> console.log('readable'); }); > >> > >> response.on('end', function () { console.log('end'); }); }, 5); > >> }); > >> > >> Here's my shell session: > >> > >> $ node -v v0.10.0 $ node http-test.js got response with status > >> 200 $ > >> > >> Is this the correct behavior? > >> > >> -- Michael Jackson @mjackson > >> > >> > >> On Thu, Mar 21, 2013 at 4:27 PM, Isaac Schlueter <[email protected] > > <mailto:[email protected]> > >> <mailto:[email protected] <mailto:[email protected]>>> wrote: > >> > >> re old-mode > >> > >> Yes, that's fine. If you just want to get all the > > data asap, use > >> on('data', handler). It'll work great, and it's still > > very fast. > >> pause()/resume(), the whole bit. (The difference is > > that it won't > >> emit data until you're listening, and pause() will > > *actually* pause.) > >> > >> > >> Re read(cb) > >> > >> It's problematic for reasons that I've discussed all > > of the places > >> where it's been brought up. That horse is dead, let's > > stop beating > >> it. (There were a few other proposals as well, btw. > > Reducibles and > >> some other monadic approaches come to mind.) > >> > >> > >> Re pipe() vs looping around read() vs custom Writable > > vs on('data') > >> > >> Whatever works for your case is fine. It's flexible > > on purpose, and > >> allows more types of consumption than streams1, and > > creating custom > >> writables is easier than it was in streams1. > >> > >> If you find something that the API can't do for you, > > or find yourself > >> doing a lot of backflips or overriding a lot of > > methods to get your > >> stuff working, then let's chat about it in a github > > issue. You might > >> be missing something, or you might have found a > > genuine shortcoming in > >> the API. > >> > >> > >> > >> On Thu, Mar 21, 2013 at 2:01 PM, Sigurgeir Jonsson > >> <[email protected] > > <mailto:[email protected]> > > <mailto:[email protected] > > <mailto:[email protected]>>> > >> wrote: > >>> Thanks for all the answers. I almost forgot to look > > back at this > >> thread as > >>> the custom writeStreams have exceeded the high > > expectation I had > >> already for > >>> Streams 2. For me, the reference manual was a little > >>> confusing, > > as there are > >> complete > >>> examples on using the read method, no mention of > > "reading" through a > >>> writeStream endpoint. > >>> > >>> Marco, I agree that that read has more detailed > > control of minimum > >> incoming > >>> content. However I wonder if it would be more > > efficient to default > >>> pipe.chunkSize to a "lowWatermark" of the receiver > > (if defined). > >> This > >>> lowWatermark could be adjusted dynamically and the > > callback in the > >> writable > >>> should keep sequence of events under control? > >>> > >>> Anyway, thanks Node team, I'm very impressed! > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> On Wednesday, March 20, 2013 4:45:32 AM UTC-4, Marco > > Rogers wrote: > >>>> > >>>> @Nathan's response is right. Creating a writable > > stream is > >> preferable in > >>>> most cases. But I wanted to add a little context to > > that. If > >> you're dealing > >>>> with a base readable stream, it's just pushing > > chunks of data at > >> you off the > >>>> wire. Your first task is to collect those chunks > > into meaningful > >> data. So > >>>> IMO the reason creating a writable stream is > > preferable is because it > >>>> prompts you not just read off the stream, but to > > create semantics > >> around > >>>> what the new stream is supposed to be. The api > > reflects this > >> opinion and > >>>> that's why creating writable streams feels like the > > more natural > >> way, and > >>>> the ugliness of dealing with read() is wrapped up > > in the pipe() > >> method. It > >>>> was kind of designed that way. > >>>> > >>>> But the read() api was also designed for a use > > case. It's meant > >> to handle > >>>> low/high water marks effectively, as well as enable > > more > >> optimized special > >>>> parsing by reading off specific lengths of chunks. > > These were > >> things that > >>>> people kept needing, but the old api didn't support > > well. If you were > >>>> writing a library for a special parser, you might > > write a custom > >> Writable > >>>> stream and inside it you would be using the read(n) > > api to > >> control *how* you > >>>> read data off the socket. I hope that makes sense. > >>>> > >>>> :Marco > >>>> > >>>> On Monday, March 18, 2013 11:06:48 AM UTC-7, > > Sigurgeir Jonsson wrote: > >>>>> > >>>>> The new streams have excellent support for > > high/low watermarks and > >>>>> auto-pausing/resuming, but the documentation > > confuses me a little... > >>>>> particularly the read method. > >>>>> > >>>>> When I read the new docs for the first time I was > > under the > >> impression > >>>>> that the optimal way to become a user of a stream > > is to write > >> loops around > >>>>> the read functio. However in practice I find > > myself simply > >> writing custom > >>>>> writeStreams and use the callback to control > > upstream pressure > >> (in addition > >>>>> to source Watermarks if needed). Here is an > > example where I > >> move the > >>>>> output to a queue that executes a custom function > > in parallel (i.e. > >>>>> uploading to a database) > > https://gist.github.com/ZJONSSON/5189249 > >>>>> > >>>>> Are there any benefits to using the read method > > directly on a > >> stream vs. > >>>>> piping to a custom Writable stream? > >>> > >>> -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: > >>> > > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > > > > > >> You received this message because you are subscribed > > to the Google > >>> Groups "nodejs" group. To post to this group, send email to > > [email protected] <mailto:[email protected]> > >> <mailto:[email protected] > > <mailto:[email protected]>> > >>> To unsubscribe from this group, send email to > >>> [email protected] > > <mailto:nodejs%[email protected]> > >> <mailto:nodejs%[email protected] > > <mailto:nodejs%[email protected]>> > >>> For more options, visit this group at > >>> http://groups.google.com/group/nodejs?hl=en?hl=en > >>> > >>> --- You received this message because you are subscribed > > to the Google > >> Groups > >>> "nodejs" group. To unsubscribe from this group and stop > >>> receiving > > emails from it, > >> send an > >>> email to [email protected] > > <mailto:nodejs%[email protected]> > >> <mailto:nodejs%[email protected] > > <mailto:nodejs%[email protected]>>. > >>> For more options, visit > > https://groups.google.com/groups/opt_out. > >>> > >>> > >> > >> -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: > >> > > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > > > > You received this message because you are subscribed to the > > Google > >> Groups "nodejs" group. To post to this group, send email to > > [email protected] <mailto:[email protected]> > >> <mailto:[email protected] > > <mailto:[email protected]>> > >> To unsubscribe from this group, send email to > >> [email protected] > > <mailto:nodejs%[email protected]> > >> <mailto:nodejs%[email protected] > > <mailto:nodejs%[email protected]>> > >> For more options, visit this group at > >> http://groups.google.com/group/nodejs?hl=en?hl=en > >> > >> --- You received this message because you are subscribed > > to the Google > >> Groups "nodejs" group. To unsubscribe from this group and stop > >> receiving > > emails from it, > >> send an email to [email protected] > > <mailto:nodejs%[email protected]> > >> <mailto:nodejs%[email protected] > > <mailto:nodejs%[email protected]>>. > >> For more options, visit > > https://groups.google.com/groups/opt_out. > >> > >> > >> > >> -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: > >> > > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > > > > You received this message because you are subscribed to the > > Google > >> Groups "nodejs" group. To post to this group, send email to > > [email protected] <mailto:[email protected]> > >> To unsubscribe from this group, send email to > >> [email protected] > > <mailto:nodejs%[email protected]> > >> For more options, visit this group at > >> http://groups.google.com/group/nodejs?hl=en?hl=en > >> > >> --- You received this message because you are subscribed to > > the Google > >> Groups "nodejs" group. To unsubscribe from this group and stop > >> receiving emails > > from it, send > >> an email to [email protected] > > <mailto:nodejs%[email protected]>. > >> For more options, visit > > https://groups.google.com/groups/opt_out. > >> > >> > > > > > > -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: > > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > > > > > You received this message because you are subscribed to the Google > > Groups "nodejs" group. To post to this group, send email to > > [email protected] <mailto:[email protected]> To > > unsubscribe from this group, send email to > > [email protected] > > <mailto:nodejs%[email protected]> For more options, > > visit this group at > > http://groups.google.com/group/nodejs?hl=en?hl=en > > > > --- You received this message because you are subscribed to the > > Google Groups "nodejs" group. To unsubscribe from this group and > > stop receiving emails from it, send an email to > > [email protected] > > <mailto:nodejs%[email protected]>. For more options, > > visit https://groups.google.com/groups/opt_out. > > > > > > > > > > -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: > > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > > > > > You received this message because you are subscribed to the Google > > Groups "nodejs" group. To post to this group, send email to > > [email protected] <mailto:[email protected]> To > > unsubscribe from this group, send email to > > [email protected] > > <mailto:nodejs%[email protected]> For more options, > > visit this group at > > http://groups.google.com/group/nodejs?hl=en?hl=en > > > > --- You received this message because you are subscribed to the > > Google Groups "nodejs" group. To unsubscribe from this group and > > stop receiving emails from it, send an email to > > [email protected] > > <mailto:nodejs%[email protected]>. For more options, > > visit https://groups.google.com/groups/opt_out. > > > > > > > > > > -- > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > > --- > You received this message because you are subscribed to the Google Groups > "nodejs" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
