Thanks all, I'll look into these solutions and see what I can come up with. Cheers
On Sat, Mar 29, 2014 at 12:41 PM, Bruno Jouhier <[email protected]> wrote: > I have all sorts of data pumps running between JSON files, mongo, mysql > and oracle. Memory usage is completely flat (I actually used it to debug > memory leaks in the Oracle driver). > > The pumps are built with https://github.com/Sage/ez-streams > > For example JSON to mongo: > > var ez = require('ez-streams'); > > function jsonFileToMongo(errHandler, sourceFileName, targetMongoCollection) { > var reader = ez.devices.file.text.reader(sourceFileName); > var writer = ez.devices.mongodb.writer(targetMongoCollection); > reader.transform(ez.transforms.json.parser()).pipe(errHandler, writer); > } > > > Ez-streams are callback-based rather than event-based. This has two > important consequences: > > - back-pressure is a non-issue. You don't control back pressure by > wiring events but by inserting .buffer(bufsize) elements in your processing > chains. > - exception handling is robust: every chains ends with a "reducer" > (pipe is a reducer). All the exceptions which are not caught explicitly in > the transforms are propagated to the reducer's error handler. > > There is no device yet for couchDB but it takes very little to write a > device (mongo device is 26 lines, oracle device 31 lines). > > Bruno > > > On Saturday, March 29, 2014 12:22:21 PM UTC+1, Floby wrote: >> >> Your approach is good but using the data event won't let you have enough >> control on the speed of the flow. >> >> I have a similar use case, only with xml files. At the end of my pipeline >> is a writable stream I called a cradlepusher because I used the cradle >> module. >> >> It's implemented from stream2 writable stream which will handle back >> pressure for you when implemented correctly. The whole point if these is >> that they're easier to get right than old streams. >> >> CradlePusher.prototype._write = function _write(resource, encoding, >> callback) { >> if(typeof resource.id === 'undefined') { >> return callback(new TypeError('Given resource does not have an ID')); >> } >> >> var self = this; >> >> this._db.get(resource.id, function(err, doc) { >> if (!err) { >> self._updateDoc(doc, resource, callback); >> } >> else if (err.error === 'not_found' && err.reason === 'missing') { >> self._pushNewDoc(resource, callback); >> } >> else { >> return callback(err); >> } >> }); >> }; >> >> My use case also has an update logic, which is why I'm not using bulk >> updates anymore and I have to get a doc before updating it. But the >> interesting part is the use of the callback of the write method. Calling it >> at the right time will get your back pressure handles more graciously. >> >> -- > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > > --- > You received this message because you are subscribed to a topic in the > Google Groups "nodejs" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/nodejs/j15sn3hg_Pc/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- Many thanks, Miles Burton -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
