On 02/26/2013 06:57 AM, Roberto De Ioris wrote: >> On 02/25/2013 01:18 AM, Roberto De Ioris wrote: >>>> On 02/24/2013 01:02 AM, Roberto De Ioris wrote: >>>>>> Hello, >>>>>> >>>>>> I am using the uWSGI cache feature to reduce the number of DB >>>>>> queries. >>>>>> The >>>>>> flow looks like this (code is written in Python): >>>>>> >>>>>> request comes in: >>>>>> >>>>>> value = cache_get(key) >>>>>> if value: >>>>>> return value >>>>>> else: >>>>>> value = db_query(...) >>>>>> cache_set(key, pickle.dumps(value), 60) >>>>>> >>>>>> It works well for few minutes. After that, it calls cache_set for >>>>>> some >>>>>> keys more than once every 60 second (once a second or more >>>>>> frequently). >>>>>> I >>>>>> would expect cache_set to not be called more than once every 60 >>>>>> seconds >>>>>> because I pass expires=60 to cache_set. >>>>>> >>>>>> cache_get doesn't return anything even if cache_set succeeds (return >>>>>> value >>>>>> is True). This is what worries me. >>>>>> >>>>>> Let me know if you need more details. >>>>>> >>>>>> uWSGI config: >>>>>> >>>>>> [uwsgi] >>>>>> socket=/var/run/uwsgi.sock >>>>>> master=true >>>>>> enable-threads=true >>>>>> listen=8192 >>>>>> socket-timeout=60 >>>>>> buffer-size=8192 >>>>>> pidfile=/var/run/uwsgi.pid >>>>>> module=app.start >>>>>> pythonpath=blabla >>>>>> processes=8 >>>>>> threads=0 >>>>>> daemonize=/var/log/uwsgi.log >>>>>> disable-logging=false >>>>>> logto=/var/log/uwsgi-access.log >>>>>> logdate=true >>>>>> no-orphans=true >>>>>> vacuum=true >>>>>> uid=user >>>>>> gid=group >>>>>> cache=10000 >>>>>> cache-blocksize=4096 >>>>>> cache-report-freed-items=true_______________________________________________ >>>>>> uWSGI mailing list >>>>>> [email protected] >>>>>> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi >>>>>> >>>>> Just two things: >>>>> >>>>> - be sure to run a version >= 1.4.4 as older version had a corner-case >>>>> bug >>>>> in which collisioned keys could be swallowed (if you have lot of >>>>> collisions you could have hit that) >>>>> >>>>> - remember that cache_set does not overwrite already-existent keys. >>>>> Use >>>>> cache_update if you want auto-overwrite. >>>>> >>>> I am using 1.4.5. I used to encounter the deadlock issue with 0.9.9.2 >>>> so >>>> I moved to the latest version recently. >>>> >>>> What I am seeing right now happens rarely but it fills up the cache >>>> (10000 elements) even if I am only trying to store about 80 keys. When >>>> that issue happens, I see cache_set being called over and over for the >>>> same key and cache_get keeps returning None. It quickly fills up the >>>> cache even if cache_set is called for the same key. >>>> >>>> cache is empty. >>>> cache_set: key1, expires 60. >>>> cache_set: key1, expires 60. >>>> cache_get: key1 -> None. >>>> cache_set: key1, expires 60. >>>> cache_get: key1 -> None. >>>> ... >>>> cache is reported as full in the log. >>>> >>>> cache_set is called for the same key from multiple workers but with the >>>> uWSGI locking mechanism, it should not be an issue and cache_get should >>>> return something after the first cache_set succeeds. >>>> >>>> I am puzzled right now. >>>> _______________________________________________ >>>> uWSGI mailing list >>>> [email protected] >>>> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi >>>> >>> Have you noted if the problem starts happening after more than an hour >>> or >>> earlier ? the only thing popping in my mind is some bug in the >>> auto-expiration system (80 keys with 60 seconds expiration should start >>> giving problems pretty late on a 10k store) >>> >> Last time it happened, it was earlier than an hour. The cache was empty >> and the workers started adding the keys. Because cache_get would not >> return anything, they continued calling cache_set over and over and at a >> rapid rate. The strange thing is that the cache got full with those >> calls even if only 80 different keys were used. >> >> To me, it seems that cache_set increased the number of elements in the >> cache but the new elements were not accessible through cache_get during >> that time. >> _______________________________________________ >> > I have tried reproducing it for hours but without luck :( > > The only thing i can think of, is some difference in the keyname you are > using (i mean, i write something but you ask for another). > > Are you using python3 ? > > May i ask you to try with uWSGI 1.9 (+ stats enabled) as it exports > internal cache statistics ? > Thanks for spending time on it.
I did some debugging and I opened a pull request on the uwsgi-1.4 branch: https://github.com/unbit/uwsgi/pull/160. It describes what I think is going on + a potential fix. Let me know your thoughts. _______________________________________________ uWSGI mailing list [email protected] http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
