Looking through your stream-parser module is looks like it internally
buffers the incoming data. I forgot to mention that the files I'm reading
are many gigabytes in size so I think I really need a pull based stream
solution. I've been able to accomplish this using syntax like:
parser.createParser(fs.createReadStream('fs')).pipe(processor). I was just
looking for a more elegant all pipe based solution.
On Tuesday, February 5, 2013 4:36:05 PM UTC-5, Nathan Rajlich wrote:
>
> Well it sounds like you want to implement a Transform stream subclass:
> http://nodejs.org/docs/v0.9.8/api/stream.html#stream_class_stream_transform
>
> The parsing you're describing (read n bytes, do some parsing, repeat)
> sounds like my "stream-parser" (
> https://github.com/TooTallNate/node-stream-parser) mixin would help you.
> So you'd mix the Parser into your Transform stream subclass' prototype, and
> then just call _bytes(n, callback); to parse incoming data that gets
> written to the transform stream.
>
> Cheers!
>
> On Tue, Feb 5, 2013 at 1:28 PM, Joe Ferner <[email protected]<javascript:>
> > wrote:
>
>> I'm trying to write a parser/Transformer using the new Stream2 API
>> (pull/read stream). I want to do something like this:
>>
>> fs.createReadStream('file').pipe(parser).pipe(processor);
>>
>>
>> I don't have control over "processor". I would like to have my parser
>> read/pull "x" number of bytes from the file stream, parse it, then emit the
>> parsed data to the next step (in this case processor). I've tried listening
>> to the "pipe" event and attaching an event handler to the source Stream's
>> "readable" event but that doesn't seem to work. The underlying node Stream
>> API tries to call my write method instead of waiting for me to read from it.
>>
>> Does the new Stream2 API support pull/readable Streams in the middle of
>> pipes?
>>
>> --
>> --
>> Job Board: http://jobs.nodejs.org/
>> Posting guidelines:
>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>> You received this message because you are subscribed to the Google
>> Groups "nodejs" group.
>> To post to this group, send email to [email protected]<javascript:>
>> To unsubscribe from this group, send email to
>> [email protected] <javascript:>
>> For more options, visit this group at
>> http://groups.google.com/group/nodejs?hl=en?hl=en
>>
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "nodejs" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected] <javascript:>.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>
>
>
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups
"nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
