I have googled around for this but haven't found much. My environment: Ubuntu 9.04 Django 1.1 beta1 cmemcache 0.95 memcached 1.2.2 python 2.6.2 Apache 2.2.11 mod_fastcgi 2.4.6
I am using the cached_db session setting in django to write through memcache to a sqlite db which will be replaced with something better later. I am benchmarking my app with a cluster using httperf and other benchmarking scripts. When serving small numbers of requests ( ~ 10 req/s ) everything works just fine. When I load it up more to 60 req/s I start seeing 500s with these messages in the log files: [[email protected]] mcm_fetch_cmd():1181: memcache(4) protocol error: Operation now in progress: server sent data for key not in request [[email protected]] mcm_fetch_cmd():1310: memcache(4) protocol error: �}qUcurrent_sessionqU$407b36b0-3b60-11de-96ba-001e4fe25c1fqs. If I load it up with more req/s the main django process crashes and I get one of these in my syslog: kernel: [ 6769.234808] python[8118] trap stack segment ip:7fbb3b4eed50 sp:7fbb2a0551a0 error:0 and more of this in the fastcgi logs: [[email protected]] mcm_fetch_cmd():1181: memcache(4) protocol error: Operation now in progress: server sent data for key not in request [[email protected]] mcm_fetch_cmd():1310: memcache(4) protocol error: �}qUcurrent_sessionqU$8094ad94-3b56-11de-96b9-001e4fe25c1fqs. [[email protected]] mcm_fetch_cmd():1181: memcache(4) protocol error: Operation now in progress: server sent data for key not in request [[email protected]] mcm_fetch_cmd():1310: memcache(4) protocol error: �}qUcurrent_sessionqU$09646924-3b5d-11de-96ba-001e4fe25c1fqs. [[email protected]] mcm_fetch_cmd():1181: memcache(4) protocol error: server sent data for key not in request [[email protected]] mcm_server_readable():2566 [[email protected]] mcm_get_line():1597 [[email protected]] mcm_server_send_cmd():2711: failed to send command to the memcache server: Operation now in progress: Operation now in progress [[email protected]] mcm_fetch_cmd():1181: memcache(4) protocol error: Operation now in progress: server sent data for key not in request [[email protected]] mcm_fetch_cmd():1310: memcache(4) protocol error: �}qUcurrent_sessionqU$407b36b0-3b60-11de-96ba-001e4fe25c1fqs. The code using the cache is very simple, I have a decorator that wraps several views: def uuid_session_decorator(request, *args, **kw): my_uuid = request.session.get('current_session',False) if not my_uuid: my_uuid = uuid.uuid1().__str__() request.session['current_session'] = my_uuid return view(request, my_uuid, *args, **kw) This is the only place that I am using sessions in my app. If I dont use the decorator function I dont see any errors, and django can handle a large amount more req/s before throwing 500s and it doesnt crash. I changed the session to cache only to rule out issues with sqlite and it has the same behavior and errors. Im only writing a uuid string to the cache so I dont see why it should be erroring out so quickly. Does anyone know what these errors mean and why it is causing django to crash? The box is a quad core with 6GB ram by the way and never hits swap during the tests.
