Wrapping streams is doable, and can be very expedient, but it can get expensive. It's better to not do this if you don't have to.
The basic idea is that, every time a new chunk of data comes in, you call stream._readableState.onread(null, chunk). Then, if stream._readableState.length > stream._readableState.highWaterMark, pause the source. Whenever some data is consumed, if the length is less than the highWaterMark, then resume the source. Definitely needs some API love. On Fri, Dec 21, 2012 at 7:12 PM, Arunoda Susiripala <[email protected]> wrote: > I should look at how net module does it. > > > On Saturday, December 22, 2012, Jake Verbaten wrote: >> >> You have to buffer. Ideally you use read-stream's internal buffer rather >> then your own buffer. >> >> If you apply backpressure you apply it like you apply it to TCP. >> net.Socket already handles back pressure. >> >> >> On Fri, Dec 21, 2012 at 4:52 PM, Arunoda Susiripala >> <[email protected]> wrote: >> >> Actually i really dont need to manually work on the steam api internal >> touching. What if i create an old stream and wrap it with the new one. >> >> Is that above make sense. Or do I have to intervene manually to drop >> packets. >> >> >> On Saturday, December 22, 2012, Isaac Schlueter wrote: >> >> Yes, this is doable, but it's a bit odd. Right now, the Socket class >> in lib/net.js is handling a similar case, since it has to consume data >> from a handle object that only has readStart() and readStop() methods. >> So, it's doing a lot of inappropriate touching of internal APIs that >> it shouldn't need to do. There does need to be a better mechanism for >> handling a pause/resume-able source, and perhaps a userland >> abstraction for handling a "just drop data if it comes too fast" kind >> of scenario. It'd be better if lib/net.js didn't have to use any >> internal stream APIs to get its job done. >> >> >> On Fri, Dec 21, 2012 at 12:32 PM, Arunoda Susiripala >> <[email protected]> wrote: >> > That is not the problem. My question is how to build redis-stream using >> > Stream2 ? >> > >> > >> > On Sat, Dec 22, 2012 at 1:55 AM, Thomas Blobaum <[email protected]> >> > wrote: >> >> >> >> If you havent seen redis-stream, it might help a bit. >> >> https://github.com/tblobaum/redis-stream >> >> >> >> var Redis = require('redis-stream') >> >> , client = new Redis('localhost', 6379, 0) >> >> , subscribe = client.stream('subscribe') >> >> >> >> subscribe.pipe(process.stdout) >> >> subscribe.write('channel-one') >> >> >> >> >> >> Thomas Blobaum >> >> https://github.com/tblobaum >> >> >> >> >> >> On Fri, Dec 21, 2012 at 2:20 PM, Arunoda Susiripala >> >> <[email protected]> wrote: >> >> > Yes I know buffering is not a good solution. >> >> > Seems like need to implement this with the old stream api. (emitting >> >> > data >> >> > events). >> >> > >> >> > >> >> > On Sat, Dec 22, 2012 at 12:41 AM, Bradley Meck >> >> > <[email protected]> >> >> > wrote: >> >> >> >> >> >> Personally I did not want to write a buffering system for in memory >> >> >> _read >> >> >> and just used a PassThrough stream for node-tart's substream parts : >> >> >> https://github.com/bmeck/node-tart/blob/master/lib/substream.js . >> >> >> I'm >> >> >> trying >> >> >> as well to find a sane way to allow internal buffering from a >> >> >> different >> >> >> stream but have not thought of an elegant solution. >> >> >> >> >> >> >> >> >> On Friday, December 21, 2012 10:07:07 AM UTC-6, Arunoda Susiripala >> >> >> wrote: >> >> >>> >> >> >>> Hi, >> >> >>> >> >> >>> I was looking at the new readable stream documentation and it said >> >> >>> that >> >> >>> we've to implement _read() method in order to implement the >> >> >>> readable >> >> >>> stream. >> >> >>> >> >> >>> But I have a problem? >> >> >>> >> >> >>> I want to create a readable stream where it's source is redis >> >> >>> subscribe(). How can I implement a readable stream for this with >> >> >>> the >> >> >>> new >> >> >>> API? >> >> >>> >> >> >>> Do I need to buffer incoming data from redis? >> >> >>> >> >> >>> >> >> >>> -- >> >> >>> Arunoda Susiripala >> >> >>> >> >> >>> @arunoda >> >> >>> https://github.com/arunoda >> >> >>> http://www.linkedin.com/in/arunoda >> >> >> >> >> >> -- >> >> >> Job Board: > > > > -- > Arunoda Susiripala > > @arunoda > https://github.com/arunoda > http://www.linkedin.com/in/arunoda > > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
