I'm having some trouble with Redis and I thought some of you could
enlighten me :)
I have a web2py instance running with several apps.
I have a Redis instance working as a cache, with maxmemory set to 20gb and
LRU eviction policy.
Every application connects to the same Redis instance.
One of those apps is the main one, from wich I perform some administrative
tasks like:
- getting a list of the cache keys of a specific app
- clearing all the cache keys of a specific app.
In order to get the list of the keys cached by a specific app I use this
custom code:
def get_cache_keys(application):
import re
result = []
regex = ':*'
prefix = 'w2p:%s' % application
cache_set = 'w2p:%s:___cache_set' % application
r = re.compile(regex)
buckets = redis_conn.smembers(cache_set)
if buckets:
keys = redis_conn.sunion(buckets)
for key in keys:
if r.match(str(key).replace(prefix, '', 1)):
result.append(key)
return result
In the other hand, if I need to clear all the cached keys of a specific
app, I get the key list (using the above funciton) and then iterate those
keys calling redis_conn.delete(key)
I thought this was ok, but today I've found a problem: *I've found a key
that was generated by an app but it isn't returned by the code above*.
To be sure, I connected to Redis from the terminal and in fact I was able
to get the key and its value, but running the code above the key isn't
listed. I'v checked the TTL of the key and it is still valid.
It's not the only case: I've found several cases.
For example, I have a web2py app called "transmedia".
>From redis-cli, I want to know the SETs (buckets) that web2py created to
store the keys of that specific app, so I run:
127.0.0.1:6379> SMEMBERS w2p:transmedia:___cache_set
1) "w2p:transmedia:___cache_set:26383112"
2) "w2p:transmedia:___cache_set:26384550"
3) "w2p:transmedia:___cache_set:26383115"
4) "w2p:transmedia:___cache_set:26383117"
5) "w2p:transmedia:___cache_set:26383436"
6) "w2p:transmedia:___cache_set:26383118"
7) "w2p:transmedia:___cache_set:26383113"
8) "w2p:transmedia:___cache_set:26383111"
9) "w2p:transmedia:___cache_set:26383495"
10) "w2p:transmedia:___cache_set:26383440"
11) "w2p:transmedia:___cache_set:26383170"
12) "w2p:transmedia:___cache_set:26383116"
Then I checked all those SETs to get all the keys stored by the app:
127.0.0.1:6379> SMEMBERS w2p:transmedia:___cache_set:26384550
1) "w2p:cache:transmedia:url_logo_movil"
2) "w2p:cache:transmedia:menu1"
3) "w2p:cache:transmedia:url_imagen_default"
4) "w2p:cache:transmedia:CT"
5) "w2p:cache:transmedia:url_imagen_logo_newsletter"
6) "w2p:cache:transmedia:html_head"
7) "w2p:cache:transmedia:menu0"
8) "w2p:cache:transmedia:TEMPLATE"
9) "w2p:cache:transmedia:url_fondo_personalizado"
10) "w2p:cache:transmedia:url_favicon"
11) "w2p:cache:transmedia:C"
12) "w2p:cache:transmedia:CONFIG"
13) "w2p:cache:transmedia:url_logo_grande"
14) "w2p:cache:transmedia:html_body"
15) "w2p:cache:transmedia:url_logo"
There we can see *15 keys*.
All the other sets are "(empty list or set)".
But here is the weird part: the application "transmedia" stored a key that
I don't see in the list:
*w2p:cache:transmedia:url_imagen_publicidad_newsletter*
Checking from redis-cli I can see that the key is still there and its TTL
is still valid:
127.0.0.1:6379> GET w2p:cache:transmedia:url_imagen_publicidad_newsletter
"\x80\x02U\x00."
127.0.0.1:6379> TTL w2p:cache:transmedia:url_imagen_publicidad_newsletter
(integer) 18711
I can confirm this discrepancy if I perform a SCAN like this:
$ redis-cli --scan --pattern w2p:cache:transmedia:*
w2p:cache:transmedia:url_logo_movil
w2p:cache:transmedia:url_favicon
w2p:cache:transmedia:url_fondo_personalizado
w2p:cache:transmedia:TEMPLATE
w2p:cache:transmedia:menu1
w2p:cache:transmedia:html_head
w2p:cache:transmedia:url_imagen_logo_newsletter
w2p:cache:transmedia:html_body
w2p:cache:transmedia:url_logo
w2p:cache:transmedia:CT
w2p:cache:transmedia:url_logo_grande
w2p:cache:transmedia:CONFIG
w2p:cache:transmedia:C
w2p:cache:transmedia:url_imagen_publicidad_newsletter
w2p:cache:transmedia:url_imagen_default
w2p:cache:transmedia:menu0
Notice the output now includes the *16 keys*, not 15.
So why isn't the key listed when scanning all the SETS (buckets) that
web2py created for the app? Is there something wrong with my code?
My main concern is about *clearing all the keys of a given app*.
Right now I can't trust that my code will delete all the keys, because the
function that gets the key list doesn't always include all the keys that
the app stored.
So another question: *is there a better way to delete all the keys cached
by an app?*
I thought I could use redis-cli from the commandline like this:
$ redis-cli --scan --pattern w2p:cache:transmedia:* | xargs redis-cli del
This would work, but in order to run it from within my main app I would
have to use subprocess.
What do you think?
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/web2py/cdb7d740-e6db-4055-b830-eb37baa60b51%40googlegroups.com.
