Wrote this at an offsite where we were discussing Hadoop as a proof of concept: https://github.com/dshaw/reduce-stream in conjunction with https://github.com/dshaw/filter-stream
Was able to boil down 60+ GB of log data to weighted geodata using these two modules. My apologies for the complete lack of docs. Hopefully the tests shed some light on this little mad science project. This needs a couple more stream patterns (stream fanout) to be truly useful, but the idea does work. Daniel Shaw @dshaw On Sat, Jun 23, 2012 at 2:08 PM, José F. Romaniello <[email protected]> wrote: > Excellent Dominic, yes! The idea of reduce ( idea, 0 code) was to make a > readable/writeable stream. For each chunk passed to the reduce, it will > execute the reduce function, when the input stream ends it will the reduce > stream will emit one and only one data event with the value of the > accumulator and it will ends. The example i wrote is totally wrong and > should be > > > var aFileStream = fs.createReadStream("foo.txt"); > aFileStream.pipe(funs.reduce( function (prev, chunk) { return prev + > chunk.match(/o/g).length; })) > .pipe(funs.map( function (total) { return " total amoun of 'o': > " + total; })) > .pipe(process.stdout); > > Not sure if is useful though > > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
