Not really that mad, thanks. I'm looking into it ;)
? 2013/7/16 18:11, Floby ??:
I you're ready to try mad science, you could check out the scuttlebutt
<https://github.com/dominictarr/scuttlebutt> family
On Monday, 15 July 2013 15:32:33 UTC+2, Jerry Liu wrote:
Hi, I have a question on a pub-sub implement using Node.js.
The scenarios is that I have a chat-room, but members transfer
lots of data. When new member joins in, not only all data should
be sent to him, but take part in data transmission. This is
actually a white-board scenarios. And is also like pub-sub.
I also considered using Redis or RabbitMQ. But these data is
usually not small, such as 100MB. I don't think it should be store
in database or something.
P2P should be a good solution, but I still not ready for it. Now I
still use C/S model for this.
In my implement, I have 3 ways of doing so.
Ideas:
// v1
var server = new net.Server();
server.on('connection', function(client) {
var s = this;
s.file = fs.openSync('/tmp/somewhere', 'a+'); // should be a
async call, but here just to simplify
// save to member-list
s.clients.push(client);
// read saved file and send to new client
var ddd = fs.readFileSync('/tmp/somewhere'); // should be a
async call, but here just to simplify
client.write(ddd);
client.on('data', function(data){
// send to all members
for (var i = s.clients.length - 1; i >= 0; i--) {
if (s.clients[i] != client) { // client doesn't send
data to itself
s.clients[i].write(data);
}
};
// write data into file
fs.writeFile('/tmp/somewhere', data);
});
// here should be more error handler and code due with
closed client. But I just simplify.
});
// v2
var server = new net.Server();
server.on('connection', function(client) {
var s = this;
// save to member-list
s.clients.push(client);
// read saved file and send to new client
var rd = fs.createReadStream('/tmp/somewhere'); // should be
a async call, but here just to simplify
rd.pipe(rd, {'end': false}); // set end to false, so that
client won't close after file sending
for (var i = s.clients.length - 1; i >= 0; i--) {
if (s.clients[i] != client) {
s.clients[i].pipe(client, {'end': false}); // all other
members pipe to new one
client.pipe(s.clients[i], {'end': false}); // the new
client pipes to everybody else
}
};
var wr = fs.createWriteStream('/tmp/somewhere');
client.pipe(wr, {'end': false});
// here should be more error handler and code due with
closed client. But I just simplified it.
// Also, I simplified the code that close these two streams.
});
// v3
// Well, v3 is different. It's even hard to use real code to
represent. Here I use some pseudo code.
var server = new net.Server();
server.file = fs.openSync('/tmp/somewhere', 'a+'); // should
be a async call, but here just to simplify
function processOne(cli, data, callback){
cli.write(data, callback);
};
function runPendingJobs(cli, file){
if (cli.pendingList.length < 1) {
return;
}
// fetch one job
var firstJob = cli.pendingList[0];
cli.pendingList.shift();
// read chunk
var d = readChunkFromFile(firstJob); // pseudo code
// process it and run next job when it ends
processOne(cli, d, runPendingJobs(cli, file));
};
server.on('connection', function(client) {
var s = this;
// save to member-list
s.clients.push(client);
// pending list is a list contains pending data info.
// This is more like a queue, so that client can receive
data chunks one by one
// won't be flooded
client.pendingList = [];
for (var i = s.clients.length - 1; i >= 0; i--) {
if (s.clients[i] != client) {
s.clients[i].on('data', function(data){
// when any data appears, write to file
writeToFile(server.file, data); // pseudo code
// record its postion
var newJob = {'start': fileEnds, 'size': data.length};
// and attach it to client's pending list.
client.pendingList.push(newJob);
// process the job queue
runPendingJobs(client, server.file);
});
}
}
// here should be more error handler and code due with
closed client. But I just simplify.
});
---
Results:
I have all these 3 version run in my server. But found that no one
is good.
v1 is a very straight way. But if clients speed are not
symmetrical, lots of Buffer stays in memory waiting to be sent.
More clients, more memory consumed. So is v2.
v3 store all data into file first, and only attaches minimal
information to a job queue. However, This method comsumes high cpu
useage. I have no idea why.
As a beginer of network programmer, here I wondered which of these
ideas is right. And does my results fit the idea? Also, what's the
best practice to this pub-sub problem?
Sorry for this VERY long email, I even don't know whether I
expressed it clealy. Thanks!
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
---
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
