Turns out it was custom code put in place by our devs for selective cache clearing. Was never meant to scale for sessions just partials. For some reason it was applied to sessions as well and we had a steady increase of traffic which caused the link saturation. We've removed the code and things are back to normal.
On Nov 23, 5:11 pm, Gabriel Sosa <[email protected]> wrote: > Wow I never tought synfony could do such thing. Are you using the last > stable release? > > Regards > > On 11/23/09, Chris F <[email protected]> wrote: > > > > > All, > > > We've narrowed the issue down to symfony's session handling. > > > It stores all sessions in a key called "_metadata" > > Every new session would have it's data appended to the value. > > > This value would grow to upwards of 15000 entries and over 1MB in > > size. > > > At a set time garbage collection would clean up the metadata entry and > > subsequently our latency and network throughput the drop to the floor. > > > Due to consistent hashing every request from our dynamic servers would > > retrieve that key/value pair from mem1 eventually saturate the NIC. > > > Our developers are working to remove the dependency on _metadata. > > > Thanks > > > Chris > > > On Nov 21, 4:43 pm, Chris F <[email protected]> wrote: > >> I have a small memcached server pool of 3 instances(m1.small) and an > >> autoscaling cloud of linux instances(4-20 c1.xlarge AWS EC2) that > >> store sessions to them using the memcache PECL extension wrapped by > >> Symfony 1.2 framework. > > >> In the past week we've had an increase in traffic which has resulted > >> in a recurring latency issue caused by memcache. > > >> We see the network throughput of our web servers slowly creep up over > >> an hour then drop suddenly. > >> Please see the links below to the zabbix graph pics. > > >> As the network throughput rises on both web and memcache servers the > >> latency rises. > >> The web traffic does not correlate directly with the dropoffs in > >> network throughput that can be seen on the web servers. Analyzing the > >> traffic using ntop showed 98% of the traffix was from memcache. > > >> It seems that the pecl memcache extension is not properly hashing the > >> keys evenly across the 3 memcache servers. In fact I had added mem3 > >> to alleviate the load with no significant gain. > >> Also the amount of memcache traffic seems highly suspect. > > >> Can someone assist in diagnosing this issue? > >> See below for additional info and links. > > >> Thanks > > >> Chris > > >> ELB Latency statistics:http://imgur.com/ZM4mh.png > >> Memcache server network utilization:http://imgur.com/Sg8fJ.png > >> PHP server network utilization:http://imgur.com/zxzWU.png > >> Memcache server CPU load:http://imgur.com/tW8La.png > >> PHP server CPU load:http://imgur.com/S5wio.png > >> WWW requests per minute:http://imgur.com/242eV.png > > >> pecl memcache settings: > >> Active persistent connections 1 > >> Version 2.2.5 > >> Revision $Revision: 1.111 $ > > >> Directive Local Value Master Value > >> memcache.allow_failover 1 1 > >> memcache.chunk_size 32768 32768 > >> memcache.default_port 11211 11211 > >> memcache.default_timeout_ms 1000 1000 > >> memcache.hash_function crc32 crc32 > >> memcache.hash_strategy consistent consistent > >> memcache.max_failover_attempts 20 20 > > >> memcache server setting: > >> memcached -d -p 11211 -u nobody -m 1024 -c 2048 -P /var/run/memcached/ > >> memcached.pid > >> v 1.4.0 > > -- > Sent from my mobile device > > Gabriel Sosa > Si buscas resultados distintos, no hagas siempre lo mismo. - Einstein
