You have multiple things there. Thousands requests (per second, I assume) is a lot even without spawning a child shell. Thousands of spawned shells, even without image magick, can bring a busy system down. But then again, you will know best with a test. Try it and see if and where you get stuck.
Even if the system itself handles it, depending on how fast you get to your saturation limits, your response time may get increasingly longer. If that's acceptable and you know there will be downtimes when the jobs can catch up, you might do well with this. The bigger problem I see there is that you cannot scale horizontally there. You wanna be able to send those convert jobs to other machines as well, probably. And for that, a simple child_process won't do, it would be better if you had some sort of a job queue (MQ, redis pubsub, something of the sort depending if you can allow failures and lost jobs). Then your service that listens for reqs only shoots them to the queue, and listens for a matching response on the response queue. And your actual png generation service can listen on the other side and only take as many requests simultaneously as it makes sense - e.g. maybe per num of CPUs or similar, test there and see what makes sense. And when you have a lot of work, just spawn another png generator on a new machine and you're good, no blocking. The best thing to do is simply measure. Try to run several processes at once and shoot a thousand requests, see how long it takes for all of them to finish. Thn try to fire 10 waves of 1k reqs, each wave 2 seconds apart, and see how that goes. (Of course, adjust the numbers to your environment.). Lastly, from my personal experience, these kinds of things regularly grow out of hand and original specs, and it's better to just do job queues from the get-go. Hope that helps. Zlatko On Thursday, May 19, 2016 at 11:51:46 PM UTC+2, Alisson Cavalcante Agiani wrote: > > I need to develop a service where the user sends an array of values, I > generate a chart with d3.js and convert it to png with a child_process > calling imagemagick to send it back to the user. Are there any caveats for > child_process when scaling for thousands of requests? > -- Job board: http://jobs.nodejs.org/ New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscr...@googlegroups.com. To post to this group, send email to nodejs@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/09ff5e6e-f2fe-45d7-bb57-488230ee78d0%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
