Check out node-canvas for the rendering, then you won't need any child processes.
https://github.com/Automattic/node-canvas On Sat, May 21, 2016 at 8:20 AM, Zlatko Đurić <zladu...@gmail.com> wrote: > You should try it. But the bottom line is, in a thousand-reqs-per-sec > scenario, you'll almost always hit a bottleneck and need to scale the > processing job over multiple machines. So it's a tradeoff - how important > is it to you to finish every request, how fast you want results, is the > 1k/sec throughput constant or peak... your imagemagick (or phantom binary, > whatever you use) can only run so many instances on a single machine. > > So again - test your load such as it is. If you already have a sample > JSON, write a node script to make a png out of it - parsing json and > calling imagemagick should not take very much time. Then run the script 1k > times in a row and see how fast it goes. Then run it in thousand parallel > shells and see if that goes well, too. > > testing, testing, testing. Hope I'm not too boring with it. > > > On Fri, May 20, 2016 at 11:33 PM, Ω Alisson <thelinuxl...@gmail.com> > wrote: > >> Interesting. I wonder if it is better to use phantomjs for the PNG >> rendering. Although it is slower, I don't need child_process >> >> On Fri, May 20, 2016 at 5:42 PM Zlatko <zladu...@gmail.com> wrote: >> >>> You have multiple things there. Thousands requests (per second, I >>> assume) is a lot even without spawning a child shell. Thousands of spawned >>> shells, even without image magick, can bring a busy system down. But then >>> again, you will know best with a test. Try it and see if and where you get >>> stuck. >>> >>> Even if the system itself handles it, depending on how fast you get to >>> your saturation limits, your response time may get increasingly longer. If >>> that's acceptable and you know there will be downtimes when the jobs can >>> catch up, you might do well with this. >>> >>> The bigger problem I see there is that you cannot scale horizontally >>> there. You wanna be able to send those convert jobs to other machines as >>> well, probably. And for that, a simple child_process won't do, it would be >>> better if you had some sort of a job queue (MQ, redis pubsub, something of >>> the sort depending if you can allow failures and lost jobs). Then your >>> service that listens for reqs only shoots them to the queue, and listens >>> for a matching response on the response queue. And your actual png >>> generation service can listen on the other side and only take as many >>> requests simultaneously as it makes sense - e.g. maybe per num of CPUs or >>> similar, test there and see what makes sense. And when you have a lot of >>> work, just spawn another png generator on a new machine and you're good, no >>> blocking. >>> >>> The best thing to do is simply measure. Try to run several processes at >>> once and shoot a thousand requests, see how long it takes for all of them >>> to finish. Thn try to fire 10 waves of 1k reqs, each wave 2 seconds apart, >>> and see how that goes. (Of course, adjust the numbers to your environment.). >>> >>> Lastly, from my personal experience, these kinds of things regularly >>> grow out of hand and original specs, and it's better to just do job queues >>> from the get-go. >>> >>> Hope that helps. >>> >>> Zlatko >>> >>> >>> On Thursday, May 19, 2016 at 11:51:46 PM UTC+2, Alisson Cavalcante >>> Agiani wrote: >>>> >>>> I need to develop a service where the user sends an array of values, I >>>> generate a chart with d3.js and convert it to png with a child_process >>>> calling imagemagick to send it back to the user. Are there any caveats for >>>> child_process when scaling for thousands of requests? >>>> >>> -- >>> Job board: http://jobs.nodejs.org/ >>> New group rules: >>> https://gist.github.com/othiym23/9886289#file-moderation-policy-md >>> Old group rules: >>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "nodejs" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to nodejs+unsubscr...@googlegroups.com. >>> To post to this group, send email to nodejs@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/nodejs/09ff5e6e-f2fe-45d7-bb57-488230ee78d0%40googlegroups.com >>> <https://groups.google.com/d/msgid/nodejs/09ff5e6e-f2fe-45d7-bb57-488230ee78d0%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- >> Job board: http://jobs.nodejs.org/ >> New group rules: >> https://gist.github.com/othiym23/9886289#file-moderation-policy-md >> Old group rules: >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "nodejs" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/nodejs/1Oi4uzV2TGw/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> nodejs+unsubscr...@googlegroups.com. >> To post to this group, send email to nodejs@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/nodejs/CACZE8Y6cqWvVTFF8Ee9SdvsBZTk-DKraTACCs_hq%2BWciTGbRcA%40mail.gmail.com >> <https://groups.google.com/d/msgid/nodejs/CACZE8Y6cqWvVTFF8Ee9SdvsBZTk-DKraTACCs_hq%2BWciTGbRcA%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Zlatko > > -- > Job board: http://jobs.nodejs.org/ > New group rules: > https://gist.github.com/othiym23/9886289#file-moderation-policy-md > Old group rules: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > --- > You received this message because you are subscribed to the Google Groups > "nodejs" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to nodejs+unsubscr...@googlegroups.com. > To post to this group, send email to nodejs@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/nodejs/CADu3pbzKt2Z85t1Of8a2GHw5VA8%3DHWzcxU%3Dm_9Nh%3Dhvx3VMBxw%40mail.gmail.com > <https://groups.google.com/d/msgid/nodejs/CADu3pbzKt2Z85t1Of8a2GHw5VA8%3DHWzcxU%3Dm_9Nh%3Dhvx3VMBxw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- Job board: http://jobs.nodejs.org/ New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscr...@googlegroups.com. To post to this group, send email to nodejs@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/CAPJ5V2YnNJ3-T6wHkGMSL7DomNbDqchQ9AM%3DP-HB5wkYSgszDQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
