[nodejs] Re: child_process and scaling
You have multiple things there. Thousands requests (per second, I assume) is a lot even without spawning a child shell. Thousands of spawned shells, even without image magick, can bring a busy system down. But then again, you will know best with a test. Try it and see if and where you get stuck. Even if the system itself handles it, depending on how fast you get to your saturation limits, your response time may get increasingly longer. If that's acceptable and you know there will be downtimes when the jobs can catch up, you might do well with this. The bigger problem I see there is that you cannot scale horizontally there. You wanna be able to send those convert jobs to other machines as well, probably. And for that, a simple child_process won't do, it would be better if you had some sort of a job queue (MQ, redis pubsub, something of the sort depending if you can allow failures and lost jobs). Then your service that listens for reqs only shoots them to the queue, and listens for a matching response on the response queue. And your actual png generation service can listen on the other side and only take as many requests simultaneously as it makes sense - e.g. maybe per num of CPUs or similar, test there and see what makes sense. And when you have a lot of work, just spawn another png generator on a new machine and you're good, no blocking. The best thing to do is simply measure. Try to run several processes at once and shoot a thousand requests, see how long it takes for all of them to finish. Thn try to fire 10 waves of 1k reqs, each wave 2 seconds apart, and see how that goes. (Of course, adjust the numbers to your environment.). Lastly, from my personal experience, these kinds of things regularly grow out of hand and original specs, and it's better to just do job queues from the get-go. Hope that helps. Zlatko On Thursday, May 19, 2016 at 11:51:46 PM UTC+2, Alisson Cavalcante Agiani wrote: > > I need to develop a service where the user sends an array of values, I > generate a chart with d3.js and convert it to png with a child_process > calling imagemagick to send it back to the user. Are there any caveats for > child_process when scaling for thousands of requests? > -- Job board: http://jobs.nodejs.org/ New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscr...@googlegroups.com. To post to this group, send email to nodejs@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/09ff5e6e-f2fe-45d7-bb57-488230ee78d0%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[nodejs] Decode String in which some escape character is there
Hello All, I am struggling to decode the string in which some escape character is present. Like this : *Googles Allo looks great but its yet another messaging app I have to deal with!. *Can anyone let me know how can I get the result without escaped character. Thanks, Anurag Mishra -- Job board: http://jobs.nodejs.org/ New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscr...@googlegroups.com. To post to this group, send email to nodejs@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/5f4287aa-07b7-4daf-93b2-8942280beccc%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[nodejs] Node.js: Basic TCP Client (à la Netcat) with Net and Readline Modules?
I'm not sure if it's bad form to repost my question from another site, but
here goes:
*My problem in short involves trying to build a very simple TCP client
(think Netcat) with node's net module.* I have had success with
sock.write('GET etc...), but when I try to use the readline module to
dynamically send data, the servers I have tried to talk to either never
respond, or don't respond how I would expect them to.
My *suspicion* is that the readline module is escaping escape characters
such as \r\n, which causes my request to the servers to be improperly
formatted. I have tried:
client.write(new Buffer(cmd))
But my basic TCP server is still interpreting /r/n as text. In fact, the
native console interprets it as text.
*Update:* I have verified by using JSON.stringify on the terminal output
that something is escaping escape characters (e.g. When I enter \r\n into
the terminal the JSON.stringify -ed terminal input reads as \\r\\n). Maybe
my terminal is doing it by default and this isn't at all a problem with
node.js? I'm using Windows Powershell.
See my client script below for further clarification:
// Inspired and sourced from:
http://www.hacksparrow.com/tcp-socket-programming-in-node-js.html
// ANSI escape code sequence for colors, etc. General Form: "\x1b[Nm" Where N
is a number.
// For http requests, see
http://stackoverflow.com/questions/6686261/what-at-the-bare-minimum-is-required-for-an-http-request
//For IRC protocol see https://news.ycombinator.com/item?id=2071318
var net = require('net');var fs = require('fs');var readline =
require('readline');
var host = 'localhost';var port = 6667;
var client = new net.Socket();
client.connect(port, host, function() {
console.log('CONNECTED TO: ' + host + ':' + port + '\n');
// Write a message to the socket as soon as the client is connected, the
server will receive it as message from the client
client.write('USER foo bar batz boo\r\nNICK test345\r\n');//
client.write('JOIN #mae\r\n');
rl = readline.createInterface({
input: process.stdin,
output: process.stdout
});
rl.on('line', function(cmd) {
var bufferCommand = new Buffer(cmd)
console.log('bufferCommand = ' + bufferCommand);
client.write(bufferCommand);
});
});
// Add a 'data' event handler for the client socket// data is what the server
sent to this socket
client.on('data', function(data) {
console.log('DATA: ' + data);
var fs = require('fs');
fs.appendFile("log.txt", data, function(err) {
if(err) {
return console.log(err);
}
console.log("The file was saved!");
});
});
// Catches error if no TCP server to answer request.
client.on('error', function(error) {
console.log(error);});
// Add a 'close' event handler for the client socket
client.on('close', function() {
console.log('Connection closed');});
All help and criticism appreciated!
--
Job board: http://jobs.nodejs.org/
New group rules:
https://gist.github.com/othiym23/9886289#file-moderation-policy-md
Old group rules:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
---
You received this message because you are subscribed to the Google Groups
"nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to nodejs+unsubscr...@googlegroups.com.
To post to this group, send email to nodejs@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/nodejs/d4f2b3c9-1f30-4bc4-9518-4174fd5e47f1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
