Hi everyone, If you have questions about RabbitMQ and Node.js, and can't find the right answer here, then do also try the rabbitmq-discuss list.
By questions I mean, on RabbitMQ behaviour or design patterns, or on node-amqp, or on rabbit.js or on SockJS. Best wishes alexis On Thu, Mar 8, 2012 at 12:45 PM, Jeroen Janssen <[email protected]> wrote: > Hi, > > I have a similar situation where I consume rabbitmq messages and > process them into a couchdb document structure. > > Currently how my solution is setup is: > *) ack enabled, prefetchCount = 1 (only one message can be pending > ack, this also results in 1 concurrent access to couchdb from my app) > *) have an "in memory" document cache > *) upon receiving a message to process I look into the cache, if it's > not there yet I fetch it from couchdb > *) updates get done to the document in cache (and then I 'ack' the > message) > *) after a little while I write everything in the cache into couchdb > (I use amqp unsubscribe to suspend incoming amqp traffic until > everything is written to couchdb) > > At the moment this solution 'solves' the 'hit couchdb everytime a > message is received' that slows processing down a lot for me. > Also it stops incoming amqp traffic while I'm actually writing data > back to couchdb (and provides a window to exit my app in a 'safe' way. > > The downside of my current implementation is that there is a window > where I could lose data (if my app crashes before writing the cache > into couchd while I have actually 'acked' the amqp message) - I > believe this is also what Chris is referring to. > > I was thinking about using amqp ack (in combination with higher > prefetchCount) until after I actually have written the cache to > couchdb. This would ensure that rabbitmq keeps the messages > (persistent) until the data is actually written into couchdb. And I > can have multiple messages 'in processing' so I don't write to couchdb > on every message being processed. > > I hope this makes sense. > > I was also wondering if anyone else has a similar setup or if there is > some kind of (better?) pattern that can be applied to this problem. > > Best regards, > > Jeroen Janssen > > On 7 mrt, 22:28, Dan Milon <[email protected]> wrote: >> Hello everyone, >> >> I'm facing the following problem. I just have set up a rabbitmq queue, >> and my node app is subscribed to that queue (via node-amqp). When a >> message arrives, the app does an insert and a few updates on a mongodb >> (via mongoose). Now the problem comes when there are say 10000 messages >> in the queue (durable), and i fire up node. It is flooded by the >> messages and although mongodb insertions are started, no callback is >> ever called. Finally all messages are consumed, and after a while >> mongoose will throw a timeout exception, or node will crash out of >> memory. During that time, mongodb reports that it received only like 30 >> queries. >> >> A temporary solution to that problem, is using rabbitmq's ACKs. That >> instructs the queue to not send another message, unless i ack the >> previous one. Now if i use that technique, although i ack just after i >> call inserts and updates (not in their callbacks), in the end things get >> throttled and with a lot less performance the app is doing what it's >> supposed to do. >> >> Anyone knows why this is happening? How to fix/avoid it? >> I'll give example code if needed. >> >> Thank you, >> Dan. > > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
