On 02/25/2013 01:18 AM, Roberto De Ioris wrote: >> On 02/24/2013 01:02 AM, Roberto De Ioris wrote: >>>> Hello, >>>> >>>> I am using the uWSGI cache feature to reduce the number of DB queries. >>>> The >>>> flow looks like this (code is written in Python): >>>> >>>> request comes in: >>>> >>>> value = cache_get(key) >>>> if value: >>>> return value >>>> else: >>>> value = db_query(...) >>>> cache_set(key, pickle.dumps(value), 60) >>>> >>>> It works well for few minutes. After that, it calls cache_set for some >>>> keys more than once every 60 second (once a second or more frequently). >>>> I >>>> would expect cache_set to not be called more than once every 60 seconds >>>> because I pass expires=60 to cache_set. >>>> >>>> cache_get doesn't return anything even if cache_set succeeds (return >>>> value >>>> is True). This is what worries me. >>>> >>>> Let me know if you need more details. >>>> >>>> uWSGI config: >>>> >>>> [uwsgi] >>>> socket=/var/run/uwsgi.sock >>>> master=true >>>> enable-threads=true >>>> listen=8192 >>>> socket-timeout=60 >>>> buffer-size=8192 >>>> pidfile=/var/run/uwsgi.pid >>>> module=app.start >>>> pythonpath=blabla >>>> processes=8 >>>> threads=0 >>>> daemonize=/var/log/uwsgi.log >>>> disable-logging=false >>>> logto=/var/log/uwsgi-access.log >>>> logdate=true >>>> no-orphans=true >>>> vacuum=true >>>> uid=user >>>> gid=group >>>> cache=10000 >>>> cache-blocksize=4096 >>>> cache-report-freed-items=true_______________________________________________ >>>> uWSGI mailing list >>>> [email protected] >>>> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi >>>> >>> Just two things: >>> >>> - be sure to run a version >= 1.4.4 as older version had a corner-case >>> bug >>> in which collisioned keys could be swallowed (if you have lot of >>> collisions you could have hit that) >>> >>> - remember that cache_set does not overwrite already-existent keys. Use >>> cache_update if you want auto-overwrite. >>> >> I am using 1.4.5. I used to encounter the deadlock issue with 0.9.9.2 so >> I moved to the latest version recently. >> >> What I am seeing right now happens rarely but it fills up the cache >> (10000 elements) even if I am only trying to store about 80 keys. When >> that issue happens, I see cache_set being called over and over for the >> same key and cache_get keeps returning None. It quickly fills up the >> cache even if cache_set is called for the same key. >> >> cache is empty. >> cache_set: key1, expires 60. >> cache_set: key1, expires 60. >> cache_get: key1 -> None. >> cache_set: key1, expires 60. >> cache_get: key1 -> None. >> ... >> cache is reported as full in the log. >> >> cache_set is called for the same key from multiple workers but with the >> uWSGI locking mechanism, it should not be an issue and cache_get should >> return something after the first cache_set succeeds. >> >> I am puzzled right now. >> _______________________________________________ >> uWSGI mailing list >> [email protected] >> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi >> > Have you noted if the problem starts happening after more than an hour or > earlier ? the only thing popping in my mind is some bug in the > auto-expiration system (80 keys with 60 seconds expiration should start > giving problems pretty late on a 10k store) > Last time it happened, it was earlier than an hour. The cache was empty and the workers started adding the keys. Because cache_get would not return anything, they continued calling cache_set over and over and at a rapid rate. The strange thing is that the cache got full with those calls even if only 80 different keys were used.
To me, it seems that cache_set increased the number of elements in the cache but the new elements were not accessible through cache_get during that time. _______________________________________________ uWSGI mailing list [email protected] http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
