All, We've narrowed the issue down to symfony's session handling.
It stores all sessions in a key called "_metadata" Every new session would have it's data appended to the value. This value would grow to upwards of 15000 entries and over 1MB in size. At a set time garbage collection would clean up the metadata entry and subsequently our latency and network throughput the drop to the floor. Due to consistent hashing every request from our dynamic servers would retrieve that key/value pair from mem1 eventually saturate the NIC. Our developers are working to remove the dependency on _metadata. Thanks Chris On Nov 21, 4:43 pm, Chris F <[email protected]> wrote: > I have a small memcached server pool of 3 instances(m1.small) and an > autoscaling cloud of linux instances(4-20 c1.xlarge AWS EC2) that > store sessions to them using the memcache PECL extension wrapped by > Symfony 1.2 framework. > > In the past week we've had an increase in traffic which has resulted > in a recurring latency issue caused by memcache. > > We see the network throughput of our web servers slowly creep up over > an hour then drop suddenly. > Please see the links below to the zabbix graph pics. > > As the network throughput rises on both web and memcache servers the > latency rises. > The web traffic does not correlate directly with the dropoffs in > network throughput that can be seen on the web servers. Analyzing the > traffic using ntop showed 98% of the traffix was from memcache. > > It seems that the pecl memcache extension is not properly hashing the > keys evenly across the 3 memcache servers. In fact I had added mem3 > to alleviate the load with no significant gain. > Also the amount of memcache traffic seems highly suspect. > > Can someone assist in diagnosing this issue? > See below for additional info and links. > > Thanks > > Chris > > ELB Latency statistics:http://imgur.com/ZM4mh.png > Memcache server network utilization:http://imgur.com/Sg8fJ.png > PHP server network utilization:http://imgur.com/zxzWU.png > Memcache server CPU load:http://imgur.com/tW8La.png > PHP server CPU load:http://imgur.com/S5wio.png > WWW requests per minute:http://imgur.com/242eV.png > > pecl memcache settings: > Active persistent connections 1 > Version 2.2.5 > Revision $Revision: 1.111 $ > > Directive Local Value Master Value > memcache.allow_failover 1 1 > memcache.chunk_size 32768 32768 > memcache.default_port 11211 11211 > memcache.default_timeout_ms 1000 1000 > memcache.hash_function crc32 crc32 > memcache.hash_strategy consistent consistent > memcache.max_failover_attempts 20 20 > > memcache server setting: > memcached -d -p 11211 -u nobody -m 1024 -c 2048 -P /var/run/memcached/ > memcached.pid > v 1.4.0
