That's actually a great idea! Em qui, 26 de mai de 2016 4:04 PM, Matt <hel...@gmail.com> escreveu:
> Check out node-canvas for the rendering, then you won't need any child > processes. > > https://github.com/Automattic/node-canvas > > On Sat, May 21, 2016 at 8:20 AM, Zlatko Đurić <zladu...@gmail.com> wrote: > >> You should try it. But the bottom line is, in a thousand-reqs-per-sec >> scenario, you'll almost always hit a bottleneck and need to scale the >> processing job over multiple machines. So it's a tradeoff - how important >> is it to you to finish every request, how fast you want results, is the >> 1k/sec throughput constant or peak... your imagemagick (or phantom binary, >> whatever you use) can only run so many instances on a single machine. >> >> So again - test your load such as it is. If you already have a sample >> JSON, write a node script to make a png out of it - parsing json and >> calling imagemagick should not take very much time. Then run the script 1k >> times in a row and see how fast it goes. Then run it in thousand parallel >> shells and see if that goes well, too. >> >> testing, testing, testing. Hope I'm not too boring with it. >> >> >> On Fri, May 20, 2016 at 11:33 PM, Ω Alisson <thelinuxl...@gmail.com> >> wrote: >> >>> Interesting. I wonder if it is better to use phantomjs for the PNG >>> rendering. Although it is slower, I don't need child_process >>> >>> On Fri, May 20, 2016 at 5:42 PM Zlatko <zladu...@gmail.com> wrote: >>> >>>> You have multiple things there. Thousands requests (per second, I >>>> assume) is a lot even without spawning a child shell. Thousands of spawned >>>> shells, even without image magick, can bring a busy system down. But then >>>> again, you will know best with a test. Try it and see if and where you get >>>> stuck. >>>> >>>> Even if the system itself handles it, depending on how fast you get to >>>> your saturation limits, your response time may get increasingly longer. If >>>> that's acceptable and you know there will be downtimes when the jobs can >>>> catch up, you might do well with this. >>>> >>>> The bigger problem I see there is that you cannot scale horizontally >>>> there. You wanna be able to send those convert jobs to other machines as >>>> well, probably. And for that, a simple child_process won't do, it would be >>>> better if you had some sort of a job queue (MQ, redis pubsub, something of >>>> the sort depending if you can allow failures and lost jobs). Then your >>>> service that listens for reqs only shoots them to the queue, and listens >>>> for a matching response on the response queue. And your actual png >>>> generation service can listen on the other side and only take as many >>>> requests simultaneously as it makes sense - e.g. maybe per num of CPUs or >>>> similar, test there and see what makes sense. And when you have a lot of >>>> work, just spawn another png generator on a new machine and you're good, no >>>> blocking. >>>> >>>> The best thing to do is simply measure. Try to run several processes at >>>> once and shoot a thousand requests, see how long it takes for all of them >>>> to finish. Thn try to fire 10 waves of 1k reqs, each wave 2 seconds apart, >>>> and see how that goes. (Of course, adjust the numbers to your >>>> environment.). >>>> >>>> Lastly, from my personal experience, these kinds of things regularly >>>> grow out of hand and original specs, and it's better to just do job queues >>>> from the get-go. >>>> >>>> Hope that helps. >>>> >>>> Zlatko >>>> >>>> >>>> On Thursday, May 19, 2016 at 11:51:46 PM UTC+2, Alisson Cavalcante >>>> Agiani wrote: >>>>> >>>>> I need to develop a service where the user sends an array of values, I >>>>> generate a chart with d3.js and convert it to png with a child_process >>>>> calling imagemagick to send it back to the user. Are there any caveats for >>>>> child_process when scaling for thousands of requests? >>>>> >>>> -- >>>> Job board: http://jobs.nodejs.org/ >>>> New group rules: >>>> https://gist.github.com/othiym23/9886289#file-moderation-policy-md >>>> Old group rules: >>>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "nodejs" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to nodejs+unsubscr...@googlegroups.com. >>>> To post to this group, send email to nodejs@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/nodejs/09ff5e6e-f2fe-45d7-bb57-488230ee78d0%40googlegroups.com >>>> <https://groups.google.com/d/msgid/nodejs/09ff5e6e-f2fe-45d7-bb57-488230ee78d0%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> -- >>> Job board: http://jobs.nodejs.org/ >>> New group rules: >>> https://gist.github.com/othiym23/9886289#file-moderation-policy-md >>> Old group rules: >>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >>> --- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "nodejs" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/nodejs/1Oi4uzV2TGw/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> nodejs+unsubscr...@googlegroups.com. >>> To post to this group, send email to nodejs@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/nodejs/CACZE8Y6cqWvVTFF8Ee9SdvsBZTk-DKraTACCs_hq%2BWciTGbRcA%40mail.gmail.com >>> <https://groups.google.com/d/msgid/nodejs/CACZE8Y6cqWvVTFF8Ee9SdvsBZTk-DKraTACCs_hq%2BWciTGbRcA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Zlatko >> >> -- >> Job board: http://jobs.nodejs.org/ >> New group rules: >> https://gist.github.com/othiym23/9886289#file-moderation-policy-md >> Old group rules: >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> --- >> You received this message because you are subscribed to the Google Groups >> "nodejs" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to nodejs+unsubscr...@googlegroups.com. >> To post to this group, send email to nodejs@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/nodejs/CADu3pbzKt2Z85t1Of8a2GHw5VA8%3DHWzcxU%3Dm_9Nh%3Dhvx3VMBxw%40mail.gmail.com >> <https://groups.google.com/d/msgid/nodejs/CADu3pbzKt2Z85t1Of8a2GHw5VA8%3DHWzcxU%3Dm_9Nh%3Dhvx3VMBxw%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > Job board: http://jobs.nodejs.org/ > New group rules: > https://gist.github.com/othiym23/9886289#file-moderation-policy-md > Old group rules: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > --- > You received this message because you are subscribed to the Google Groups > "nodejs" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to nodejs+unsubscr...@googlegroups.com. > To post to this group, send email to nodejs@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/nodejs/CAPJ5V2YnNJ3-T6wHkGMSL7DomNbDqchQ9AM%3DP-HB5wkYSgszDQ%40mail.gmail.com > <https://groups.google.com/d/msgid/nodejs/CAPJ5V2YnNJ3-T6wHkGMSL7DomNbDqchQ9AM%3DP-HB5wkYSgszDQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- Job board: http://jobs.nodejs.org/ New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscr...@googlegroups.com. To post to this group, send email to nodejs@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/CACZE8Y7C0SSKNiaSLUNhWGhOp6dqj2BqeX9ZjrGCKYTOPy%2Bj_Q%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
