Ok I have created simple gist to test this. https://gist.github.com/xdenser/8944752
Just run file and you should see WTF! 49152 WTF! 114688 WTF! 180224 WTF! 245760 ... WTF! 4112384 writeStream finish WTF! 4177920 read Stream Finished if you see just writeStream finish read Stream Finished it works as expected. what I made is two "simultanous" streams 1. writes growing numbers into file and slows down from time to time 2. my custom read stream implementation to catch on that written file WTF! mean that it read a number which is less than previous. And that is not possible as numbers are growing. Somebody can explain,please, what is wrong with my CatchStream? tested on node v0.10.24 and v0.10.16 under W7 2014-02-11 13:37 GMT+02:00 greelgorke <[email protected]>: > In Addition to that, whenthe counter can emit something > > > counter._transform = function(chunk, encoding, callback){ > count += (chunk && chunk.length) || 0 > this.push(chunk) > if(count >= threshhold) this.emit('threshhold_reached') > callback() > } > > > Am Dienstag, 11. Februar 2014 12:36:20 UTC+1 schrieb greelgorke: > >> you can count bytes with by putting a passthrough stream in between >> >> var counter = new stream.Transform >> var count = 0 >> counter._transform = function(chunk, encoding, callback){ >> count += (chunk && chunk.length) || 0 >> this.push(chunk) >> callback() >> } >> >> >> slowSourceStream.pipe(counter).pipe(destinationStream) >> >> >> >> Am Samstag, 8. Februar 2014 01:12:37 UTC+1 schrieb Denys Khanzhiyev: >>> >>> Hello, >>> >>> I have a task where one slow stream is piped to fs.writeStream and after >>> some event I need to read from that writen file, i.e. read from growing >>> file from some position. A have seen `node-growing-file`, and >>> `tailing-stream` nothing seems to solve my problem. >>> >>> It looks like I do not understand how streams work >>> Here is my helper object (though it is called PxyStream it is not stream >>> in fact), >>> >>> var >>> fs = require('fs'); >>> >>> function PxyStream(path,readStream,writeStream,start,end){ >>> this.path = path; >>> this.readStream = readStream; >>> this.writeStream = writeStream; >>> this._offset = start; >>> this.endPos = end; >>> this.writeStream.on('finish',function(){ >>> this._writeStreamFinished = true; >>> this.nextStream(); >>> }.bind(this)) >>> } >>> >>> >>> PxyStream.prototype.pipe = function(destination){ >>> this.destination = destination; >>> this.nextStream(); >>> } >>> >>> PxyStream.prototype.nextStream = function(){ >>> if(!this._stream){ >>> var options = { >>> start: this._offset >>> }; >>> var last = this._writeStreamFinished; >>> console.log('new read stream',this._offset, last); >>> this._stream = fs.createReadStream(this.path,options); >>> this._stream.pipe(this.destination,{end: false}); >>> this._stream.on('data',function(data){ >>> this._offset += data.length; >>> }.bind(this)); >>> >>> this._stream.on('end',function(){ >>> console.log('read stream end',this._offset, last); >>> this._stream.unpipe(); >>> this._stream = null; >>> if(last) { >>> this.destination.emit('end'); >>> } >>> this._watch(); >>> }.bind(this)); >>> } >>> } >>> >>> PxyStream.prototype._watch = function(){ >>> this.readStream.once('data',function(){ >>> this.nextStream(); >>> }.bind(this)) >>> } >>> >>> exports.PxyStream = PxyStream; >>> >>> >>> I am using it as >>> >>> pxyStream = new PxyStream(filePath,<slowReadStream>, >>> <fsWriteStream>,start,null); >>> // i need end position too but lets skip it for now >>> pxyStream.pipe(<otherSlowStream>); >>> >>> my problem is I see 'read stream end' message far before otherSlowStream >>> ends. >>> In fact it never ends, but i can see its progress. >>> Actually destination is http.response stream. >>> I thought stream.pipe should slow down reading in order to keep buffers >>> short. >>> Maybe attached 'data' event makes it read fast, but how can I count >>> bytes then? >>> The other problem is I can not end destination properly. >>> >>> >>> >>> -- > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > > --- > You received this message because you are subscribed to the Google Groups > "nodejs" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
