Thats great effort Dustin in removing the redundancy. Coming back, my simple test is that the key-miss counts should all be *zero*, because, I've earlier stored objects for all keys in memcache. But, here the key-miss counts are *not* zero proving that my purpose is not served.
Before proceeding, let me explain my requirement clearly. I want to maintain a huge hash table to store some rules (keys) and their counts (values). The hash table has to be accessed over the network by multiple processes running parallely, which would update the count (add the incremental count to the existing value) for the rules, if they are already present in the hash table. The first time a rule is seen, it would be 'set' to the hash with the initial count as value. Once, I finish the process, I would want to write down the final counts for the individual rules in a file. I thought that memcahed could be used as a shared hash table, whose values are to be updated simultaneously by multiple processes. I tried setting an expiry time of 60*60*24*3 (3 days) while storing the objects, so that they will not be removed for making way for other objects. But, this doesn't seem to work. Now, my question is: is memcache not expected to store the objects at least till the specified expiry time? I tried using a dbm-hash sometime back and found that it is *very* slow (coz, my hash table size is going to be huge having above 20 million rules). Hoping that there is some solution for my problem :-) - B On Wed, Mar 18, 2009 at 6:47 PM, Dustin <[email protected]> wrote: > > > I didn't see any sets in there at all. Can you make a simple and > complete test that demonstrates the failure to achieve your desired > behavior? > > I'd like to rule out a bug in your program, although I'm thinking > you may just want memcached to do things it doesn't want to do. > > BTW, I got rid of a bit of redundancy in there (can't run it, though > as it's not quite complete): > > def xtract_rCounts(rcountFile, falignFile, ralignFile): > '''Extracts the rule counts for the key and write in a file''' > > key_misses = [0, 0, 0] > transforms = ['', '|||F', '|||R'] > files = [open(fn, 'w') for fn in [rcountFile, falignFile, > ralignFile]] > > for src in keyDoD.keys(): > src_key = src.replace(' ', '-%-') > for tgt in keyDoD[src].keys(): > tgt_key = tgt.replace(' ', '-%-') > > for (transform, stat, f) in zip(transforms, range(3), > files): > mc_key = tgt_key + transform > > count = mc.get(mc_key) > if count: > f.write("%s ||| %s ||| %s\n" % (src, tgt, > r_count)) > else: > key_misses[stat] += 1 > print len(mc_key), mc_key > > for f in files: > f.close() > > print "Total # of key misses for rule count : %d" % (key_misses > [1]) > print "Total # of key misses for fwrd align : %d" % (key_misses > [2]) > print "Total # of key misses for rvrs align : %d" % (key_misses > [3]) > > On Mar 18, 6:27 pm, Baskaran Sankaran <[email protected]> wrote: > > On Wed, Mar 18, 2009 at 6:22 PM, Dustin <[email protected]> wrote: > > > > > On Mar 18, 6:12 pm, Baskaran <[email protected]> wrote: > > > > > > As I understand the -M option is going to ensure that all the values > > > > that I store in memcached will be retained, unless I delete them > > > > explicitly. > > > > > -M means objects won't be prematurely removed from memcache -- I'm > > > not completely sure how safe it is for general operation. > > > > > If your goal is to build a permanent memory storage, you're > > > deviating from general use. > > > > > > As you can see, more than half the entries are missing and I can't > > > > figure out why? Has someone got some idea? > > > > > It'd be more useful to see what your client actually does. Can you > > > post a simple test? > > >>> Ok, I have stored all the keys in a separate file, when they were > added > > > > to memcached and my python script now reads the keys and get the values. > The > > relevant snippet is below: > > > > def xtract_rCounts(rcountFile, falignFile, ralignFile): > > '''Extracts the rule counts for the key and write in a file''' > > > > key_misses1 = 0 > > key_misses2 = 0 > > key_misses3 = 0 > > rC = open(rcountFile, 'w') > > fA = open(falignFile, 'w') > > rA = open(ralignFile, 'w') > > for src in keyDoD.keys(): > > src_key = src.replace(' ', '-%-') > > for tgt in keyDoD[src].keys(): > > tgt_key = tgt.replace(' ', '-%-') > > mc_key = src_key + '|||' + tgt_key > > mc_keyF = mc_key + '|||F' > > mc_keyR = mc_key + '|||R' > > > > r_count = mc.get(mc_key) > > if r_count is None: > > key_misses1 += 1 > > print len(mc_key), mc_key > > else: rC.write( "%s ||| %s ||| %s\n" % (src, tgt, r_count) ) > > > > f_align = mc.get(mc_keyF) > > if f_align is None: > > key_misses2 += 1 > > print len(mc_keyF), mc_keyF > > else: fA.write( "%s ||| %s ||| %s\n" % (src, tgt, f_align) ) > > > > r_align = mc.get(mc_keyR) > > if r_align is None: > > key_misses3 += 1 > > print len(mc_keyR), mc_keyR > > else: rA.write( "%s ||| %s ||| %s\n" % (src, tgt, r_align) ) > > > > rC.close() > > fA.close() > > rA.close() > > print( "Total # of key misses for rule count : %d" % (key_misses1) ) > > print( "Total # of key misses for fwrd align : %d" % (key_misses2) ) > > print( "Total # of key misses for rvrs align : %d" % (key_misses3) ) > > > > - B >
