It only buffers for `n` bytes when you call _bytes(n, cb). Usually that's a small payload like a frame header or similar. If you're just passing through a lot of bytes untouched then you call the _passthrough(n, cb) function, where absolutely no buffering happens.
On Wed, Feb 6, 2013 at 6:26 AM, Joe Ferner <[email protected]> wrote: > Looking through your stream-parser module is looks like it internally > buffers the incoming data. I forgot to mention that the files I'm reading > are many gigabytes in size so I think I really need a pull based stream > solution. I've been able to accomplish this using syntax like: > parser.createParser(fs.createReadStream('fs')).pipe(processor). I was just > looking for a more elegant all pipe based solution. > > > On Tuesday, February 5, 2013 4:36:05 PM UTC-5, Nathan Rajlich wrote: > >> Well it sounds like you want to implement a Transform stream subclass: >> http://nodejs.org/**docs/v0.9.8/api/stream.html#** >> stream_class_stream_transform<http://nodejs.org/docs/v0.9.8/api/stream.html#stream_class_stream_transform> >> >> The parsing you're describing (read n bytes, do some parsing, repeat) >> sounds like my "stream-parser" (https://github.com/** >> TooTallNate/node-stream-parser<https://github.com/TooTallNate/node-stream-parser> >> **) mixin would help you. So you'd mix the Parser into your Transform >> stream subclass' prototype, and then just call _bytes(n, callback); to >> parse incoming data that gets written to the transform stream. >> >> Cheers! >> >> On Tue, Feb 5, 2013 at 1:28 PM, Joe Ferner <[email protected]> wrote: >> >>> I'm trying to write a parser/Transformer using the new Stream2 API >>> (pull/read stream). I want to do something like this: >>> >>> fs.createReadStream('file').**pipe(parser).pipe(processor); >>> >>> >>> I don't have control over "processor". I would like to have my parser >>> read/pull "x" number of bytes from the file stream, parse it, then emit the >>> parsed data to the next step (in this case processor). I've tried listening >>> to the "pipe" event and attaching an event handler to the source Stream's >>> "readable" event but that doesn't seem to work. The underlying node Stream >>> API tries to call my write method instead of waiting for me to read from it. >>> >>> Does the new Stream2 API support pull/readable Streams in the middle of >>> pipes? >>> >>> -- >>> -- >>> Job Board: http://jobs.nodejs.org/ >>> Posting guidelines: https://github.com/joyent/**node/wiki/Mailing-List-* >>> *Posting-Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>> You received this message because you are subscribed to the Google >>> Groups "nodejs" group. >>> To post to this group, send email to [email protected] >>> >>> To unsubscribe from this group, send email to >>> nodejs+un...@**googlegroups.com >>> >>> For more options, visit this group at >>> http://groups.google.com/**group/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>> >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "nodejs" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to nodejs+un...@**googlegroups.com. >>> >>> For more options, visit >>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >>> . >>> >>> >>> >> >> -- > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > > --- > You received this message because you are subscribed to the Google Groups > "nodejs" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
