Hi! On Thu, Feb 16, 2012 at 9:20 PM, Dean Landolt <[email protected]> wrote: > > > On Thu, Feb 16, 2012 at 1:48 PM, HG <[email protected]> wrote: >> >> Hi! >> >> On Thu, Feb 16, 2012 at 3:38 PM, Dean Landolt <[email protected]> >> wrote: >> > >> > >> > On Thu, Feb 16, 2012 at 7:09 AM, HG <[email protected]> wrote: >> >> >> >> Hi! >> >> >> >> I created a small counting algorithm first in JavaScript. Seems to >> >> work ok. But this is of course single threaded and I could only store >> >> snapshots of it to DB. So I started thinking that what if I use Redis >> >> directly as the memory. Then I could have multiple threads or even >> >> servers running against it. >> >> >> >> However, the algorithm limits the stuff that is kept in memory. So, >> >> basically I need to do something like this: >> >> >> >> redis.zincrby(key, 1, str, function (err, reply) { >> >> redis.zcard(key, function (err, N) { >> >> If (N > maxSize) { >> >> // do stuff with the keys, including delete some (this is a >> >> few callbacks here...) >> >> } >> >> } >> >> I.e. I add counters (zincrby ands new one if it doesn't exists), but >> >> if there are too many as a result of that, I need to do some cleaning. >> >> >> >> But obviously, the whole thing works in threaded situation only if >> >> that code is blocking. So, I can't run multiple servers against redis >> >> like this or this whole code should block redis (I guess I could store >> >> this kind of procedure to MongoDB...). But is there even a way to >> >> block single node.js event loop for this? I know that would not be >> >> wise (for keeping up the speed), but I'm just trying to learn how to >> >> do this kind of stuff? Should all cases where I need blocking with the >> >> DB done with stored procedures in the DB? >> > >> > >> > >> > Technically you can do this with a blocking child process call, but I'm >> > pretty sure you don't want to do that. Can you be a little more specific >> > with what you're trying to do? Are you looking for a priority queue? >> >> I'm keeping count of the most frequent strings that I get as input. >> Basically, this is the main part of the JS implementation that I >> thought I could do with redis: >> >> var top = function (size) { >> var items = []; >> var counts = []; >> >> return { >> newItem: function (str) { >> n++; >> var ind = items.indexOf(str); >> if (ind >= 0){ >> // boost the one >> counts[ind] += 1; >> } else { >> // not there, so add it >> if (items.length < size){ >> items.push(str); >> counts.push(1); >> } else { >> // too many, lower counts and remove if zero >> for (var i = 0; i < items.length; i++) { >> counts[i] -= 1; >> if (counts[i] === 0){ >> counts.splice(i,1); >> items.splice(i,1); >> i -= 1; >> } >> } >> } >> } >> }, >> ... >> >> And in the input loop, I call this with top.newItem(input_str). But >> with this implementation, I can only support one thread. > > > > You want a lowest-first priority queue. Check github, I'm sure there's at > least a few implementations available.
I'm not sure what you mean. I'm pretty ok with the algorithm I have up there. I know it's not the best but it works ok. > But what specifically do you mean by "thread" here? Do you mean "process", > or are you doing something fancy w/in c/c++land? Regardless, you'll have to > serialize access somehow. Sorry, for not being clear. I mean single node vs. node cluster. Anyways, thinking about this more, I think it's probable that using redis to store the counters for all hits is so much slower that getting it to support cluster will not make it much faster. MongoDB definitely not as it's still executed as JavaScript there too. So I guess my assumption was wrong in the beginning. So never mind. I'll just run on single node process, it's fast enough for now anyhow. -- HG. -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
