How did this get so bloody complicated?
On Mon, Mar 25, 2013 at 3:48 PM, Michael Jackson <[email protected]>wrote: > I can see what you're saying, but the node docs do > say<http://nodejs.org/api/stream.html#stream_readable_read_size_1>that if you > don't pass a size argument to stream.read then the entire > contents of the internal buffer are returned. > > In any case, this would all be a lot easier if the readable event were > guaranteed to fire when a new readable listener is registered for the first > time. > > -- > Michael Jackson > @mjackson > > > On Mon, Mar 25, 2013 at 3:20 PM, Dan Milon <[email protected]> wrote: > >> That's not guaranteed to work. >> >> You're assuming that `stream.read()` will return the whole internal >> buffer, which is not documented anywhere. >> >> The right approach is to call `.read()` until it returns null. >> Something like that: >> >> function collectStream(stream, cb) { >> var bufs = [] >> >> function read() { >> var chunk >> >> while ((chunk = stream.read()) != null) { >> bufs.push(chunk) >> } >> } >> >> stream.on('error', cb) >> >> stream.on('readable', read) >> >> stream.on('end', function () { >> cb(null, Buffer.concat(bufs)) >> }) >> >> read() >> } >> >> On 03/25/13 23:55, Michael Jackson wrote: >> > Ok, that makes sense. >> > >> > So the readable event is more of an advisory event. The docs >> > should probably say something about how you could possibly miss the >> > event entirely if you're doing some other IO before you try and >> > read from the stream. >> > >> > For posterity's sake, I adjusted my previous example: >> > >> > var http = require('http'); >> > >> > http.get('http://www.google.com', function (response) { >> > console.log('got response with status ' + response.statusCode); >> > >> > setTimeout(function () { bufferStream(response, function (err, >> > buffer) { console.log(buffer.toString()); }); }, 1000); }); >> > >> > function bufferStream(stream, callback) { var chunks = []; >> > >> > var chunk = stream.read(); if (chunk) { chunks.push(chunk); } >> > >> > stream.on('readable', function () { chunks.push(stream.read()); >> > }); >> > >> > stream.on('error', function (error) { callback(error); }); >> > >> > stream.on('end', function () { callback(null, >> > Buffer.concat(chunks)); }); } >> > >> > You can use the bufferStream function to catch all data on the >> > stream, no matter how far in the future you are. >> > >> > -- Michael Jackson @mjackson >> > >> > >> > On Mon, Mar 25, 2013 at 1:49 PM, Dean Landolt >> > <[email protected] <mailto:[email protected]>> wrote: >> > >> > You can always call `stream.read`, at any time. This is how data >> > is /pulled/ off the stream (instead of it being pushed to you, >> > whether you're ready or not). Because of this you won't lose any >> > data. With new streams there's no real notion of a paused state -- >> > it's always paused. Once you grok that it may not seem so >> > counter-intuitive. >> > >> > The `readable` event is like a corollary to `drain` -- there to >> > tell you that it's worth bothering with a call to read. You don't >> > /have/ to listen for it -- a (needlessly inefficient) stream reader >> > could just as easily poll stream.read for new data periodically. >> > >> > >> > On Mon, Mar 25, 2013 at 4:42 PM, Michael Jackson >> > <[email protected] <mailto:[email protected]>> wrote: >> > >> > readable is emitted after you've actually started reading. >> > >> > >> > That's not what it says in the docs >> > <http://nodejs.org/api/stream.html#stream_event_readable>. >> > >> > ### Event: 'readable' When there is data ready to be consumed, this >> > event will fire. When this event emits, call the read() method to >> > consume the data. ### >> > >> > Calling stream.read *before* you get the "readable" event is >> > totally counterintuitive. >> > >> > -- Michael Jackson @mjackson >> > >> > In your example, you dont ever `response.read()`, so no readable >> > event is ever emitted. >> > >> > As you said, streams start in paused state and ready to be read. >> > >> > On 03/25/13 22:28, Michael Jackson wrote: >> >> Is it correct to assume that a Readable won't emit the >> > "readable" event >> >> until you're registered for it? >> >> >> >> Reading through the streams2 docs, I was under the >> > impression that all >> >> streams start out paused and don't start emitting data >> > until you add >> >> either a "data" (for old streams) or a "readable" >> > listener. For new >> >> streams, this should mean that they don't emit "readable" >> > until at least >> >> one listener is registered. Otherwise we still need to do >> > some buffering >> >> in order to capture all the data. >> >> >> >> For example, this code misses the readable event on node 0.10: >> >> >> >> var http = require('http'); >> >> >> >> http.get('http://www.google.com', function (response) { >> >> console.log('got response with status ' + >> > response.statusCode); >> >> >> >> setTimeout(function () { response.on('readable', function () { >> >> console.log('readable'); }); >> >> >> >> response.on('end', function () { console.log('end'); }); }, 5); >> >> }); >> >> >> >> Here's my shell session: >> >> >> >> $ node -v v0.10.0 $ node http-test.js got response with status >> >> 200 $ >> >> >> >> Is this the correct behavior? >> >> >> >> -- Michael Jackson @mjackson >> >> >> >> >> >> On Thu, Mar 21, 2013 at 4:27 PM, Isaac Schlueter <[email protected] >> > <mailto:[email protected]> >> >> <mailto:[email protected] <mailto:[email protected]>>> wrote: >> >> >> >> re old-mode >> >> >> >> Yes, that's fine. If you just want to get all the >> > data asap, use >> >> on('data', handler). It'll work great, and it's still >> > very fast. >> >> pause()/resume(), the whole bit. (The difference is >> > that it won't >> >> emit data until you're listening, and pause() will >> > *actually* pause.) >> >> >> >> >> >> Re read(cb) >> >> >> >> It's problematic for reasons that I've discussed all >> > of the places >> >> where it's been brought up. That horse is dead, let's >> > stop beating >> >> it. (There were a few other proposals as well, btw. >> > Reducibles and >> >> some other monadic approaches come to mind.) >> >> >> >> >> >> Re pipe() vs looping around read() vs custom Writable >> > vs on('data') >> >> >> >> Whatever works for your case is fine. It's flexible >> > on purpose, and >> >> allows more types of consumption than streams1, and >> > creating custom >> >> writables is easier than it was in streams1. >> >> >> >> If you find something that the API can't do for you, >> > or find yourself >> >> doing a lot of backflips or overriding a lot of >> > methods to get your >> >> stuff working, then let's chat about it in a github >> > issue. You might >> >> be missing something, or you might have found a >> > genuine shortcoming in >> >> the API. >> >> >> >> >> >> >> >> On Thu, Mar 21, 2013 at 2:01 PM, Sigurgeir Jonsson >> >> <[email protected] >> > <mailto:[email protected]> >> > <mailto:[email protected] >> > <mailto:[email protected]>>> >> >> wrote: >> >>> Thanks for all the answers. I almost forgot to look >> > back at this >> >> thread as >> >>> the custom writeStreams have exceeded the high >> > expectation I had >> >> already for >> >>> Streams 2. For me, the reference manual was a little >> >>> confusing, >> > as there are >> >> complete >> >>> examples on using the read method, no mention of >> > "reading" through a >> >>> writeStream endpoint. >> >>> >> >>> Marco, I agree that that read has more detailed >> > control of minimum >> >> incoming >> >>> content. However I wonder if it would be more >> > efficient to default >> >>> pipe.chunkSize to a "lowWatermark" of the receiver >> > (if defined). >> >> This >> >>> lowWatermark could be adjusted dynamically and the >> > callback in the >> >> writable >> >>> should keep sequence of events under control? >> >>> >> >>> Anyway, thanks Node team, I'm very impressed! >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> On Wednesday, March 20, 2013 4:45:32 AM UTC-4, Marco >> > Rogers wrote: >> >>>> >> >>>> @Nathan's response is right. Creating a writable >> > stream is >> >> preferable in >> >>>> most cases. But I wanted to add a little context to >> > that. If >> >> you're dealing >> >>>> with a base readable stream, it's just pushing >> > chunks of data at >> >> you off the >> >>>> wire. Your first task is to collect those chunks >> > into meaningful >> >> data. So >> >>>> IMO the reason creating a writable stream is >> > preferable is because it >> >>>> prompts you not just read off the stream, but to >> > create semantics >> >> around >> >>>> what the new stream is supposed to be. The api >> > reflects this >> >> opinion and >> >>>> that's why creating writable streams feels like the >> > more natural >> >> way, and >> >>>> the ugliness of dealing with read() is wrapped up >> > in the pipe() >> >> method. It >> >>>> was kind of designed that way. >> >>>> >> >>>> But the read() api was also designed for a use >> > case. It's meant >> >> to handle >> >>>> low/high water marks effectively, as well as enable >> > more >> >> optimized special >> >>>> parsing by reading off specific lengths of chunks. >> > These were >> >> things that >> >>>> people kept needing, but the old api didn't support >> > well. If you were >> >>>> writing a library for a special parser, you might >> > write a custom >> >> Writable >> >>>> stream and inside it you would be using the read(n) >> > api to >> >> control *how* you >> >>>> read data off the socket. I hope that makes sense. >> >>>> >> >>>> :Marco >> >>>> >> >>>> On Monday, March 18, 2013 11:06:48 AM UTC-7, >> > Sigurgeir Jonsson wrote: >> >>>>> >> >>>>> The new streams have excellent support for >> > high/low watermarks and >> >>>>> auto-pausing/resuming, but the documentation >> > confuses me a little... >> >>>>> particularly the read method. >> >>>>> >> >>>>> When I read the new docs for the first time I was >> > under the >> >> impression >> >>>>> that the optimal way to become a user of a stream >> > is to write >> >> loops around >> >>>>> the read functio. However in practice I find >> > myself simply >> >> writing custom >> >>>>> writeStreams and use the callback to control >> > upstream pressure >> >> (in addition >> >>>>> to source Watermarks if needed). Here is an >> > example where I >> >> move the >> >>>>> output to a queue that executes a custom function >> > in parallel (i.e. >> >>>>> uploading to a database) >> > https://gist.github.com/ZJONSSON/5189249 >> >>>>> >> >>>>> Are there any benefits to using the read method >> > directly on a >> >> stream vs. >> >>>>> piping to a custom Writable stream? >> >>> >> >>> -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: >> >>> >> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> > >> > >> >> You received this message because you are subscribed >> > to the Google >> >>> Groups "nodejs" group. To post to this group, send email to >> > [email protected] <mailto:[email protected]> >> >> <mailto:[email protected] >> > <mailto:[email protected]>> >> >>> To unsubscribe from this group, send email to >> >>> [email protected] >> > <mailto:nodejs%[email protected]> >> >> <mailto:nodejs%[email protected] >> > <mailto:nodejs%[email protected]>> >> >>> For more options, visit this group at >> >>> http://groups.google.com/group/nodejs?hl=en?hl=en >> >>> >> >>> --- You received this message because you are subscribed >> > to the Google >> >> Groups >> >>> "nodejs" group. To unsubscribe from this group and stop >> >>> receiving >> > emails from it, >> >> send an >> >>> email to [email protected] >> > <mailto:nodejs%[email protected]> >> >> <mailto:nodejs%[email protected] >> > <mailto:nodejs%[email protected]>>. >> >>> For more options, visit >> > https://groups.google.com/groups/opt_out. >> >>> >> >>> >> >> >> >> -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: >> >> >> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> > >> > You received this message because you are subscribed to the >> > Google >> >> Groups "nodejs" group. To post to this group, send email to >> > [email protected] <mailto:[email protected]> >> >> <mailto:[email protected] >> > <mailto:[email protected]>> >> >> To unsubscribe from this group, send email to >> >> [email protected] >> > <mailto:nodejs%[email protected]> >> >> <mailto:nodejs%[email protected] >> > <mailto:nodejs%[email protected]>> >> >> For more options, visit this group at >> >> http://groups.google.com/group/nodejs?hl=en?hl=en >> >> >> >> --- You received this message because you are subscribed >> > to the Google >> >> Groups "nodejs" group. To unsubscribe from this group and stop >> >> receiving >> > emails from it, >> >> send an email to [email protected] >> > <mailto:nodejs%[email protected]> >> >> <mailto:nodejs%[email protected] >> > <mailto:nodejs%[email protected]>>. >> >> For more options, visit >> > https://groups.google.com/groups/opt_out. >> >> >> >> >> >> >> >> -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: >> >> >> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> > >> > You received this message because you are subscribed to the >> > Google >> >> Groups "nodejs" group. To post to this group, send email to >> > [email protected] <mailto:[email protected]> >> >> To unsubscribe from this group, send email to >> >> [email protected] >> > <mailto:nodejs%[email protected]> >> >> For more options, visit this group at >> >> http://groups.google.com/group/nodejs?hl=en?hl=en >> >> >> >> --- You received this message because you are subscribed to >> > the Google >> >> Groups "nodejs" group. To unsubscribe from this group and stop >> >> receiving emails >> > from it, send >> >> an email to [email protected] >> > <mailto:nodejs%[email protected]>. >> >> For more options, visit >> > https://groups.google.com/groups/opt_out. >> >> >> >> >> > >> > >> > -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: >> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> > >> > >> You received this message because you are subscribed to the Google >> > Groups "nodejs" group. To post to this group, send email to >> > [email protected] <mailto:[email protected]> To >> > unsubscribe from this group, send email to >> > [email protected] >> > <mailto:nodejs%[email protected]> For more options, >> > visit this group at >> > http://groups.google.com/group/nodejs?hl=en?hl=en >> > >> > --- You received this message because you are subscribed to the >> > Google Groups "nodejs" group. To unsubscribe from this group and >> > stop receiving emails from it, send an email to >> > [email protected] >> > <mailto:nodejs%[email protected]>. For more options, >> > visit https://groups.google.com/groups/opt_out. >> > >> > >> > >> > >> > -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: >> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> > >> > >> You received this message because you are subscribed to the Google >> > Groups "nodejs" group. To post to this group, send email to >> > [email protected] <mailto:[email protected]> To >> > unsubscribe from this group, send email to >> > [email protected] >> > <mailto:nodejs%[email protected]> For more options, >> > visit this group at >> > http://groups.google.com/group/nodejs?hl=en?hl=en >> > >> > --- You received this message because you are subscribed to the >> > Google Groups "nodejs" group. To unsubscribe from this group and >> > stop receiving emails from it, send an email to >> > [email protected] >> > <mailto:nodejs%[email protected]>. For more options, >> > visit https://groups.google.com/groups/opt_out. >> > >> > >> > >> > >> >> -- >> -- >> Job Board: http://jobs.nodejs.org/ >> Posting guidelines: >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> You received this message because you are subscribed to the Google >> Groups "nodejs" group. >> To post to this group, send email to [email protected] >> To unsubscribe from this group, send email to >> [email protected] >> For more options, visit this group at >> http://groups.google.com/group/nodejs?hl=en?hl=en >> >> --- >> You received this message because you are subscribed to the Google Groups >> "nodejs" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > -- > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > > --- > You received this message because you are subscribed to the Google Groups > "nodejs" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
