That's not guaranteed to work.
You're assuming that `stream.read()` will return the whole internal
buffer, which is not documented anywhere.
The right approach is to call `.read()` until it returns null.
Something like that:
function collectStream(stream, cb) {
var bufs = []
function read() {
var chunk
while ((chunk = stream.read()) != null) {
bufs.push(chunk)
}
}
stream.on('error', cb)
stream.on('readable', read)
stream.on('end', function () {
cb(null, Buffer.concat(bufs))
})
read()
}
On 03/25/13 23:55, Michael Jackson wrote:
> Ok, that makes sense.
>
> So the readable event is more of an advisory event. The docs
> should probably say something about how you could possibly miss the
> event entirely if you're doing some other IO before you try and
> read from the stream.
>
> For posterity's sake, I adjusted my previous example:
>
> var http = require('http');
>
> http.get('http://www.google.com', function (response) {
> console.log('got response with status ' + response.statusCode);
>
> setTimeout(function () { bufferStream(response, function (err,
> buffer) { console.log(buffer.toString()); }); }, 1000); });
>
> function bufferStream(stream, callback) { var chunks = [];
>
> var chunk = stream.read(); if (chunk) { chunks.push(chunk); }
>
> stream.on('readable', function () { chunks.push(stream.read());
> });
>
> stream.on('error', function (error) { callback(error); });
>
> stream.on('end', function () { callback(null,
> Buffer.concat(chunks)); }); }
>
> You can use the bufferStream function to catch all data on the
> stream, no matter how far in the future you are.
>
> -- Michael Jackson @mjackson
>
>
> On Mon, Mar 25, 2013 at 1:49 PM, Dean Landolt
> <[email protected] <mailto:[email protected]>> wrote:
>
> You can always call `stream.read`, at any time. This is how data
> is /pulled/ off the stream (instead of it being pushed to you,
> whether you're ready or not). Because of this you won't lose any
> data. With new streams there's no real notion of a paused state --
> it's always paused. Once you grok that it may not seem so
> counter-intuitive.
>
> The `readable` event is like a corollary to `drain` -- there to
> tell you that it's worth bothering with a call to read. You don't
> /have/ to listen for it -- a (needlessly inefficient) stream reader
> could just as easily poll stream.read for new data periodically.
>
>
> On Mon, Mar 25, 2013 at 4:42 PM, Michael Jackson
> <[email protected] <mailto:[email protected]>> wrote:
>
> readable is emitted after you've actually started reading.
>
>
> That's not what it says in the docs
> <http://nodejs.org/api/stream.html#stream_event_readable>.
>
> ### Event: 'readable' When there is data ready to be consumed, this
> event will fire. When this event emits, call the read() method to
> consume the data. ###
>
> Calling stream.read *before* you get the "readable" event is
> totally counterintuitive.
>
> -- Michael Jackson @mjackson
>
> In your example, you dont ever `response.read()`, so no readable
> event is ever emitted.
>
> As you said, streams start in paused state and ready to be read.
>
> On 03/25/13 22:28, Michael Jackson wrote:
>> Is it correct to assume that a Readable won't emit the
> "readable" event
>> until you're registered for it?
>>
>> Reading through the streams2 docs, I was under the
> impression that all
>> streams start out paused and don't start emitting data
> until you add
>> either a "data" (for old streams) or a "readable"
> listener. For new
>> streams, this should mean that they don't emit "readable"
> until at least
>> one listener is registered. Otherwise we still need to do
> some buffering
>> in order to capture all the data.
>>
>> For example, this code misses the readable event on node 0.10:
>>
>> var http = require('http');
>>
>> http.get('http://www.google.com', function (response) {
>> console.log('got response with status ' +
> response.statusCode);
>>
>> setTimeout(function () { response.on('readable', function () {
>> console.log('readable'); });
>>
>> response.on('end', function () { console.log('end'); }); }, 5);
>> });
>>
>> Here's my shell session:
>>
>> $ node -v v0.10.0 $ node http-test.js got response with status
>> 200 $
>>
>> Is this the correct behavior?
>>
>> -- Michael Jackson @mjackson
>>
>>
>> On Thu, Mar 21, 2013 at 4:27 PM, Isaac Schlueter <[email protected]
> <mailto:[email protected]>
>> <mailto:[email protected] <mailto:[email protected]>>> wrote:
>>
>> re old-mode
>>
>> Yes, that's fine. If you just want to get all the
> data asap, use
>> on('data', handler). It'll work great, and it's still
> very fast.
>> pause()/resume(), the whole bit. (The difference is
> that it won't
>> emit data until you're listening, and pause() will
> *actually* pause.)
>>
>>
>> Re read(cb)
>>
>> It's problematic for reasons that I've discussed all
> of the places
>> where it's been brought up. That horse is dead, let's
> stop beating
>> it. (There were a few other proposals as well, btw.
> Reducibles and
>> some other monadic approaches come to mind.)
>>
>>
>> Re pipe() vs looping around read() vs custom Writable
> vs on('data')
>>
>> Whatever works for your case is fine. It's flexible
> on purpose, and
>> allows more types of consumption than streams1, and
> creating custom
>> writables is easier than it was in streams1.
>>
>> If you find something that the API can't do for you,
> or find yourself
>> doing a lot of backflips or overriding a lot of
> methods to get your
>> stuff working, then let's chat about it in a github
> issue. You might
>> be missing something, or you might have found a
> genuine shortcoming in
>> the API.
>>
>>
>>
>> On Thu, Mar 21, 2013 at 2:01 PM, Sigurgeir Jonsson
>> <[email protected]
> <mailto:[email protected]>
> <mailto:[email protected]
> <mailto:[email protected]>>>
>> wrote:
>>> Thanks for all the answers. I almost forgot to look
> back at this
>> thread as
>>> the custom writeStreams have exceeded the high
> expectation I had
>> already for
>>> Streams 2. For me, the reference manual was a little
>>> confusing,
> as there are
>> complete
>>> examples on using the read method, no mention of
> "reading" through a
>>> writeStream endpoint.
>>>
>>> Marco, I agree that that read has more detailed
> control of minimum
>> incoming
>>> content. However I wonder if it would be more
> efficient to default
>>> pipe.chunkSize to a "lowWatermark" of the receiver
> (if defined).
>> This
>>> lowWatermark could be adjusted dynamically and the
> callback in the
>> writable
>>> should keep sequence of events under control?
>>>
>>> Anyway, thanks Node team, I'm very impressed!
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Wednesday, March 20, 2013 4:45:32 AM UTC-4, Marco
> Rogers wrote:
>>>>
>>>> @Nathan's response is right. Creating a writable
> stream is
>> preferable in
>>>> most cases. But I wanted to add a little context to
> that. If
>> you're dealing
>>>> with a base readable stream, it's just pushing
> chunks of data at
>> you off the
>>>> wire. Your first task is to collect those chunks
> into meaningful
>> data. So
>>>> IMO the reason creating a writable stream is
> preferable is because it
>>>> prompts you not just read off the stream, but to
> create semantics
>> around
>>>> what the new stream is supposed to be. The api
> reflects this
>> opinion and
>>>> that's why creating writable streams feels like the
> more natural
>> way, and
>>>> the ugliness of dealing with read() is wrapped up
> in the pipe()
>> method. It
>>>> was kind of designed that way.
>>>>
>>>> But the read() api was also designed for a use
> case. It's meant
>> to handle
>>>> low/high water marks effectively, as well as enable
> more
>> optimized special
>>>> parsing by reading off specific lengths of chunks.
> These were
>> things that
>>>> people kept needing, but the old api didn't support
> well. If you were
>>>> writing a library for a special parser, you might
> write a custom
>> Writable
>>>> stream and inside it you would be using the read(n)
> api to
>> control *how* you
>>>> read data off the socket. I hope that makes sense.
>>>>
>>>> :Marco
>>>>
>>>> On Monday, March 18, 2013 11:06:48 AM UTC-7,
> Sigurgeir Jonsson wrote:
>>>>>
>>>>> The new streams have excellent support for
> high/low watermarks and
>>>>> auto-pausing/resuming, but the documentation
> confuses me a little...
>>>>> particularly the read method.
>>>>>
>>>>> When I read the new docs for the first time I was
> under the
>> impression
>>>>> that the optimal way to become a user of a stream
> is to write
>> loops around
>>>>> the read functio. However in practice I find
> myself simply
>> writing custom
>>>>> writeStreams and use the callback to control
> upstream pressure
>> (in addition
>>>>> to source Watermarks if needed). Here is an
> example where I
>> move the
>>>>> output to a queue that executes a custom function
> in parallel (i.e.
>>>>> uploading to a database)
> https://gist.github.com/ZJONSSON/5189249
>>>>>
>>>>> Are there any benefits to using the read method
> directly on a
>> stream vs.
>>>>> piping to a custom Writable stream?
>>>
>>> -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines:
>>>
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>
>
>> You received this message because you are subscribed
> to the Google
>>> Groups "nodejs" group. To post to this group, send email to
> [email protected] <mailto:[email protected]>
>> <mailto:[email protected]
> <mailto:[email protected]>>
>>> To unsubscribe from this group, send email to
>>> [email protected]
> <mailto:nodejs%[email protected]>
>> <mailto:nodejs%[email protected]
> <mailto:nodejs%[email protected]>>
>>> For more options, visit this group at
>>> http://groups.google.com/group/nodejs?hl=en?hl=en
>>>
>>> --- You received this message because you are subscribed
> to the Google
>> Groups
>>> "nodejs" group. To unsubscribe from this group and stop
>>> receiving
> emails from it,
>> send an
>>> email to [email protected]
> <mailto:nodejs%[email protected]>
>> <mailto:nodejs%[email protected]
> <mailto:nodejs%[email protected]>>.
>>> For more options, visit
> https://groups.google.com/groups/opt_out.
>>>
>>>
>>
>> -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines:
>>
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>
> You received this message because you are subscribed to the
> Google
>> Groups "nodejs" group. To post to this group, send email to
> [email protected] <mailto:[email protected]>
>> <mailto:[email protected]
> <mailto:[email protected]>>
>> To unsubscribe from this group, send email to
>> [email protected]
> <mailto:nodejs%[email protected]>
>> <mailto:nodejs%[email protected]
> <mailto:nodejs%[email protected]>>
>> For more options, visit this group at
>> http://groups.google.com/group/nodejs?hl=en?hl=en
>>
>> --- You received this message because you are subscribed
> to the Google
>> Groups "nodejs" group. To unsubscribe from this group and stop
>> receiving
> emails from it,
>> send an email to [email protected]
> <mailto:nodejs%[email protected]>
>> <mailto:nodejs%[email protected]
> <mailto:nodejs%[email protected]>>.
>> For more options, visit
> https://groups.google.com/groups/opt_out.
>>
>>
>>
>> -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines:
>>
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>
> You received this message because you are subscribed to the
> Google
>> Groups "nodejs" group. To post to this group, send email to
> [email protected] <mailto:[email protected]>
>> To unsubscribe from this group, send email to
>> [email protected]
> <mailto:nodejs%[email protected]>
>> For more options, visit this group at
>> http://groups.google.com/group/nodejs?hl=en?hl=en
>>
>> --- You received this message because you are subscribed to
> the Google
>> Groups "nodejs" group. To unsubscribe from this group and stop
>> receiving emails
> from it, send
>> an email to [email protected]
> <mailto:nodejs%[email protected]>.
>> For more options, visit
> https://groups.google.com/groups/opt_out.
>>
>>
>
>
> -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>
>
You received this message because you are subscribed to the Google
> Groups "nodejs" group. To post to this group, send email to
> [email protected] <mailto:[email protected]> To
> unsubscribe from this group, send email to
> [email protected]
> <mailto:nodejs%[email protected]> For more options,
> visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>
> --- You received this message because you are subscribed to the
> Google Groups "nodejs" group. To unsubscribe from this group and
> stop receiving emails from it, send an email to
> [email protected]
> <mailto:nodejs%[email protected]>. For more options,
> visit https://groups.google.com/groups/opt_out.
>
>
>
>
> -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>
>
You received this message because you are subscribed to the Google
> Groups "nodejs" group. To post to this group, send email to
> [email protected] <mailto:[email protected]> To
> unsubscribe from this group, send email to
> [email protected]
> <mailto:nodejs%[email protected]> For more options,
> visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>
> --- You received this message because you are subscribed to the
> Google Groups "nodejs" group. To unsubscribe from this group and
> stop receiving emails from it, send an email to
> [email protected]
> <mailto:nodejs%[email protected]>. For more options,
> visit https://groups.google.com/groups/opt_out.
>
>
>
>
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups
"nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
