I'm running web2py in production and I use a Redis server to store a couple
of millons sessions, but I'm facing a memory problem that I haven't been
able to fix.
I use session_expiry=172800 (two days). The application load is very stable
(it handles about 60 requests per second). I thought that after a few weeks
running I would know how much memory I need for Redis. However the memory
used by Redis keeps increasing indefinitely.
I don't use sessions for anything more than a very small percentage of
users that can login and do some administrative tasks. The vast majority of
users can't login.
Just in case, I checked and every key in redis has an expiration time:
$ redis-cli info keyspace
# Keyspace
db0:keys=1549547,expires=1548249,avg_ttl=89380135
I've also checked a few session keys and I saw that a session ocuppies
about 250 bytes. However, the memory used by Redis grows slowly and
constantly: in a whole year it reached the 24 gigabytes of RAM that the
server has, which is insane, right?
I had Redis configured to limit the amount of RAM it can use:
maxmemory 20gb
maxmemory-policy volatile-lru
However, as I commented before, after a whole year it reached that limit,
and my app started throwing this error:
Traceback (most recent call last): File "/var/www/medios/gluon/main.py",
line 462, in wsgibase session._try_store_in_db(request, response) File
"/var/www/medios/gluon/globals.py", line 1226, in _try_store_in_db
record_id = table.insert(**dd) File
"/var/www/medios/gluon/contrib/redis_session.py", line 138, in insert newid
= str(self.db.r_server.incr(self.serial)) File
"/var/www/medios/venv_medios/lib/python2.7/site-packages/redis/client.py",
line 651, in incr return self.execute_command('INCRBY', name, amount) File
"/var/www/medios/venv_medios/lib/python2.7/site-packages/redis/client.py",
line 394, in execute_command return self.parse_response(connection,
command_name, **options) File
"/var/www/medios/venv_medios/lib/python2.7/site-packages/redis/client.py",
line 404, in parse_response response = connection.read_response() File
"/var/www/medios/venv_medios/lib/python2.7/site-packages/redis/connection.py",
line 316, in read_response raise response ResponseError: OOM command not
allowed when used memory > 'maxmemory'.
I thought that error was impossible giving that Redis has a maxmemory limit
and it is instructed to evict keys when the limit is reached. However I
realised that these types of scenarios (lot of keys being written and also
lot of keys being deleted) can lead to memory fragmentation. And redis has
a defragmentation option, so I made some changes.
I reduced the maxmemory Redis limit and activated the auto defragmentation:
maxmemory 1gb
maxmemory-policy volatile-lru
activedefrag yes
Redis auto defragmentation works like a charm: when it wasn't active I
could see that the mem_fragmentation_ratio was slowly increasing. After
activating it, it stayed in a stable and optimal value of 1.05.
After a week running (remember all the sessions expire in two days) Redis
was using about 600mb of RAM. But the usage kept growing and reached the
maxmemory limit a few days later.
At that point, I could verify that Redis started evicting keys to make
space (that was expected accordingly to the configuration).
However a few days later my apps again started to throw the error with the
exact same traceback I posted before :/
What could be happening? I'm pretty sure that I don't need more than 1 or
2gb of RAM for handling the sessions with Redis. So why does it crash?
Could it be a memory leak in gluon/contrib/redis_session.py adapter?
One thing: I've never run sessions2trash.py
But if I understand the documentation
<http://web2py.com/books/default/chapter/29/13?search=cache#Sessions-in-Redis>
right, I don't need to run it as I set an expiration time for every session.
Let me know what you think, any help will be much appreciated.
Thanks!
Warm regards,
Lisandro
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/web2py/9775cb12-e932-4223-aedd-f1fd5feae1f6n%40googlegroups.com.
