The problem seemed to go away when switching to the pure python python- memcached client. There must be some thread concurrency issue in the cmemcache client that is appearing under heavy load.
On May 7, 5:00 pm, "[email protected]" <[email protected]> wrote: > I have googled around for this but haven't found much. > > My environment: > Ubuntu 9.04 > Django 1.1 beta1 > cmemcache 0.95 > memcached 1.2.2 > python 2.6.2 > Apache 2.2.11 > mod_fastcgi 2.4.6 > > I am using the cached_db session setting in django to write through > memcache to a sqlite db which will be replaced with something better > later. I am benchmarking my app with a cluster using httperf and > other benchmarking scripts. When serving small numbers of requests > ( ~ 10 req/s ) everything works just fine. When I load it up more to > 60 req/s I start seeing 500s with these messages in the log files: > > [[email protected]] mcm_fetch_cmd():1181: memcache(4) protocol > error: Operation now in progress: server sent data for key not in > request > [[email protected]] mcm_fetch_cmd():1310: memcache(4) protocol > error: }qUcurrent_sessionqU$407b36b0-3b60-11de-96ba-001e4fe25c1fqs. > > If I load it up with more req/s the main django process crashes and I > get one of these in my syslog: > > kernel: [ 6769.234808] python[8118] trap stack segment ip:7fbb3b4eed50 > sp:7fbb2a0551a0 error:0 > > and more of this in the fastcgi logs: > > [[email protected]] mcm_fetch_cmd():1181: memcache(4) protocol > error: Operation now in progress: server sent data for key not in > request > [[email protected]] mcm_fetch_cmd():1310: memcache(4) protocol > error: }qUcurrent_sessionqU$8094ad94-3b56-11de-96b9-001e4fe25c1fqs. > [[email protected]] mcm_fetch_cmd():1181: memcache(4) protocol > error: Operation now in progress: server sent data for key not in > request > [[email protected]] mcm_fetch_cmd():1310: memcache(4) protocol > error: }qUcurrent_sessionqU$09646924-3b5d-11de-96ba-001e4fe25c1fqs. > [[email protected]] mcm_fetch_cmd():1181: memcache(4) protocol > error: server sent data for key not in request > [[email protected]] mcm_server_readable():2566 > [[email protected]] mcm_get_line():1597 > [[email protected]] mcm_server_send_cmd():2711: failed to send > command to the memcache server: Operation now in progress: Operation > now in progress > [[email protected]] mcm_fetch_cmd():1181: memcache(4) protocol > error: Operation now in progress: server sent data for key not in > request > [[email protected]] mcm_fetch_cmd():1310: memcache(4) protocol > error: }qUcurrent_sessionqU$407b36b0-3b60-11de-96ba-001e4fe25c1fqs. > > The code using the cache is very simple, I have a decorator that wraps > several views: > > def uuid_session_decorator(request, *args, **kw): > my_uuid = request.session.get('current_session',False) > if not my_uuid: > my_uuid = uuid.uuid1().__str__() > request.session['current_session'] = my_uuid > return view(request, my_uuid, *args, **kw) > > This is the only place that I am using sessions in my app. If I dont > use the decorator function I dont see any errors, and django can > handle a large amount more req/s before throwing 500s and it doesnt > crash. I changed the session to cache only to rule out issues with > sqlite and it has the same behavior and errors. Im only writing a > uuid string to the cache so I dont see why it should be erroring out > so quickly. Does anyone know what these errors mean and why it is > causing django to crash? The box is a quad core with 6GB ram by the > way and never hits swap during the tests.
