At my last company we had good success with coffee-resque backed by Redis with a little custom coding around it. Each worker was an independent process that was managed and kept alive with node-foreman and monit. Jobs added to the queue included a timeout that the worker used to trigger timeouts along with handling uncaught exceptions. Not a cookie cutter system, but it kept things simple by gluing together a bunch of known good modules.
-- Daniel R. <[email protected]> [http://danielr.neophi.com/] On Wed, Dec 25, 2013 at 9:23 PM, Tim Shnaider <[email protected]> wrote: > Hi Node.js experts, > > There are many many modules and frameworks able to solve this problem. > I'm looking for CONCRETE recommendations and guidance for well supported > active implementations. > e.g. pubsub, message queues etc > > I'm a bit overwhelmed by the options and amount of research testing I'd > have to do so hoping the community can help me narrow down the choices. > > Usual scenario: > > * Front end (FE) server receives request > * FE submits task/job > * Available worker pulls task or is notifed, executes, returns result > * Result can be any JSON response > * FE notified of response/result via Promise/Callback > > Objective: > * Keep FE responsive focused on executing cheap operations to serve FE. > * Implement architecture where tasks do not execute within a timeout can > be terminated. > > Worker architecture: > * Scale across multiple servers e.g. I just need to add more physical nodes > * Configurable execution timeout resulting in termination/restart of > worker process and returning a Timeout style notification > * Will execute static JS file i.e. NOT dynamic task code > * Resilient - handles worker process failure elegantly and restart/spawn > more. > > I guess a common implementation will be a Worker Master running on a > server, > spawning X number of children that monitor a queue (pull) or are pushed a > task via sockets/pipes or some other mechanism. > > It will be up to the Master on each physical node to monitor child worker > processes and spawn more as needed. > > Thanks, > Tim > > > > > > > -- > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > > --- > You received this message because you are subscribed to the Google Groups > "nodejs" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
