> On 02/26/2013 06:57 AM, Roberto De Ioris wrote: >>> On 02/25/2013 01:18 AM, Roberto De Ioris wrote: >>>>> On 02/24/2013 01:02 AM, Roberto De Ioris wrote: >>>>>>> Hello, >>>>>>> >>>>>>> I am using the uWSGI cache feature to reduce the number of DB >>>>>>> queries. >>>>>>> The >>>>>>> flow looks like this (code is written in Python): >>>>>>> >>>>>>> request comes in: >>>>>>> >>>>>>> value = cache_get(key) >>>>>>> if value: >>>>>>> return value >>>>>>> else: >>>>>>> value = db_query(...) >>>>>>> cache_set(key, pickle.dumps(value), 60) >>>>>>> >>>>>>> It works well for few minutes. After that, it calls cache_set for >>>>>>> some >>>>>>> keys more than once every 60 second (once a second or more >>>>>>> frequently). >>>>>>> I >>>>>>> would expect cache_set to not be called more than once every 60 >>>>>>> seconds >>>>>>> because I pass expires=60 to cache_set. >>>>>>> >>>>>>> cache_get doesn't return anything even if cache_set succeeds >>>>>>> (return >>>>>>> value >>>>>>> is True). This is what worries me. >>>>>>> >>>>>>> Let me know if you need more details. >>>>>>> >>>>>>> uWSGI config: >>>>>>> >>>>>>> [uwsgi] >>>>>>> socket=/var/run/uwsgi.sock >>>>>>> master=true >>>>>>> enable-threads=true >>>>>>> listen=8192 >>>>>>> socket-timeout=60 >>>>>>> buffer-size=8192 >>>>>>> pidfile=/var/run/uwsgi.pid >>>>>>> module=app.start >>>>>>> pythonpath=blabla >>>>>>> processes=8 >>>>>>> threads=0 >>>>>>> daemonize=/var/log/uwsgi.log >>>>>>> disable-logging=false >>>>>>> logto=/var/log/uwsgi-access.log >>>>>>> logdate=true >>>>>>> no-orphans=true >>>>>>> vacuum=true >>>>>>> uid=user >>>>>>> gid=group >>>>>>> cache=10000 >>>>>>> cache-blocksize=4096 >>>>>>> cache-report-freed-items=true_______________________________________________ >>>>>>> uWSGI mailing list >>>>>>> [email protected] >>>>>>> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi >>>>>>> >>>>>> Just two things: >>>>>> >>>>>> - be sure to run a version >= 1.4.4 as older version had a >>>>>> corner-case >>>>>> bug >>>>>> in which collisioned keys could be swallowed (if you have lot of >>>>>> collisions you could have hit that) >>>>>> >>>>>> - remember that cache_set does not overwrite already-existent keys. >>>>>> Use >>>>>> cache_update if you want auto-overwrite. >>>>>> >>>>> I am using 1.4.5. I used to encounter the deadlock issue with 0.9.9.2 >>>>> so >>>>> I moved to the latest version recently. >>>>> >>>>> What I am seeing right now happens rarely but it fills up the cache >>>>> (10000 elements) even if I am only trying to store about 80 keys. >>>>> When >>>>> that issue happens, I see cache_set being called over and over for >>>>> the >>>>> same key and cache_get keeps returning None. It quickly fills up the >>>>> cache even if cache_set is called for the same key. >>>>> >>>>> cache is empty. >>>>> cache_set: key1, expires 60. >>>>> cache_set: key1, expires 60. >>>>> cache_get: key1 -> None. >>>>> cache_set: key1, expires 60. >>>>> cache_get: key1 -> None. >>>>> ... >>>>> cache is reported as full in the log. >>>>> >>>>> cache_set is called for the same key from multiple workers but with >>>>> the >>>>> uWSGI locking mechanism, it should not be an issue and cache_get >>>>> should >>>>> return something after the first cache_set succeeds. >>>>> >>>>> I am puzzled right now. >>>>> _______________________________________________ >>>>> uWSGI mailing list >>>>> [email protected] >>>>> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi >>>>> >>>> Have you noted if the problem starts happening after more than an hour >>>> or >>>> earlier ? the only thing popping in my mind is some bug in the >>>> auto-expiration system (80 keys with 60 seconds expiration should >>>> start >>>> giving problems pretty late on a 10k store) >>>> >>> Last time it happened, it was earlier than an hour. The cache was empty >>> and the workers started adding the keys. Because cache_get would not >>> return anything, they continued calling cache_set over and over and at >>> a >>> rapid rate. The strange thing is that the cache got full with those >>> calls even if only 80 different keys were used. >>> >>> To me, it seems that cache_set increased the number of elements in the >>> cache but the new elements were not accessible through cache_get during >>> that time. >>> _______________________________________________ >>> >> I have tried reproducing it for hours but without luck :( >> >> The only thing i can think of, is some difference in the keyname you are >> using (i mean, i write something but you ask for another). >> >> Are you using python3 ? >> >> May i ask you to try with uWSGI 1.9 (+ stats enabled) as it exports >> internal cache statistics ? >> > Thanks for spending time on it. > > I did some debugging and I opened a pull request on the uwsgi-1.4 > branch: https://github.com/unbit/uwsgi/pull/160. It describes what I > think is going on + a potential fix. Let me know your thoughts. >
Working on that right now, thanks a lot -- Roberto De Ioris http://unbit.it _______________________________________________ uWSGI mailing list [email protected] http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
