> Hi everyone, > > Thanks a lot for your replies. I'll try to comment upon your feedback. > > * Running several instances of Memcached sounds a bit silly indeed. I was > testing a lot of different things and somehow it was working better than > having one Memcached per server. That was a long time ago and I haven't tried > to rollback to just one instance, nor do I know how this can impact > (if any) performance. > > * The "issue" is that there is data that I can't cache into APC, I need it to > be in one place and allow all the web servers to fetch it there. And > the bottleneck of my application seems to be Memcache. At peak it can be > about 20,000 HTTP requests per second. Each of them generating one Memcache > call. If that call hangs for just a few milliseconds too much, then NGINX is > very quickly overloaded as the workers are waiting for PHP to finish > and Nginx simply maxout its workers. > > The LAN is 1Gbs. > > You are right regarding serializing. It makes it quite slower. I have also > tried gzcompress to use less network bandwidth and that takes up a lot of > CPU resource as well. > > In summary, there might not be any issue at all. But I thought Memcache > should be faster to respond. If I log all the Memcache GET that my boxes are > doing, I can see times ranging from 1ms up to 200ms or more. > > My sysadmin skills are limited, I'm a programmer. I'll try to have someone > look at our hardware and see if that can be a network issue or something.
The response times should be consistent unless something is bottlenecking in the OS or network. Have you carefully worked through the timeouts page on the wiki? You're supposed to leave that tool running for a long time to see if it can catch a lag bubble. If it can't, your issue may be elsewhere. I've pounded memcached with benchmarks of 500,000 gets/sec or more for hours on end without detecting a noticable lag bubble. That's given an environment I'm sure is clean, and much smaller request sizes (otherwise the 1gbps network is saturated too easily) The bottom line is that if you're raising the performance ceiling, you'll eventually need to break out a sysadmin hat. memcached does very well at itself serving requests in O(1) time. Keep at it and you'll figure out where the bottleneck is. Graph your network, ensure nothing's swapping, ensure your NIC can actually keep up, check dmesg for scary socket/memory errors, etc. -Dormando
