On May 21, 2014, at 5:06 AM, Mouli Kumar <[email protected]> wrote: > I am working on a NodeJs based application primarily a Web app written using > AngularJs/HTML/CSS. > I have a need for running a worker process in my server which will process > background jobs. Like, sending emails to 100s of staff at the end of the day, > or processing billing related records etc. Sometimes I also want this process > to listen to the main NodeJS process that is serving the webui, so that the > main process can handover the work to the worker for immediate processing. > > Things I have considered > 1) Worker process will run as another NodeJs app (entirely different app, no > forking, no clustering from the main process). > 2) Worker process will open a server socket for listening to the main nodejs > process for any immediate job delegation. > 3) Worker process will poll the database (mongodb) periodically to pick up > unprocessed records (say, every 5 sec) > > Your expert suggestions needed > 1) Is the approach explained above viable? > 2) Communicating between the 2 nodejs processes using sockets, is it a > recommended approach. I may have to move this worker process to a different > server and may even scale out if required. > 3) Is mongodb a good option for implementing a workload queue. My web > application uses MongoDB as database, so I did not consider moving to a > different database for the queue. Any suggestions for in-memory queue.
These are good options. As you may know, Cluster provides a built-in channel for the master to communicate with the workers but since you are looking to scale out that wouldn't be the path for you. What about job resilience, persistence, etc? What happens when there's no worker responding? A database-based solution might serve you better if these issues are important. Redis, Couch, RethinkDB (and others) all provide in-memory+persistence capabilities if you are looking for alternatives. Then there are the tools built specifically for job spooling, e.g., beankstalkd (http://kr.github.io/beanstalkd/), Kue (http://learnboost.github.io/kue/), etc. --ravi -- Job board: http://jobs.nodejs.org/ New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/48AEE4D2-573D-49FA-A325-232D858136FE%40g8o.net. For more options, visit https://groups.google.com/d/optout.
