We upgraded to 16 memcached servers using them as a standard array and now it works like a charm. Cheers, Ivo
On Oct 1, 10:18 am, ivo_m <[email protected]> wrote: > Hello Brian, > We are using PHP, with memcached_connect. Should we try switching to > pconnect? > Cheers, > Ivo > > On Sep 30, 6:34 pm, Brian Moon <[email protected]> wrote: > > > memcached is not making up connections. Something is connecting to it. > > > What is your platform? PHP? Perl? Python? Java? > > > Do you use persistent connections? > > > Brian. > > --------http://brian.moonspot.net/ > > > On 9/30/09 8:38 AM, eeyore wrote: > > > > Hello all, > > > I participate in managing a very large site (in alexa's top 70). > > > Unfortunately due to NDA I can't reveal its name. > > > We get about 8 mil unique visits per day and currently memcached is > > > the only thing that keeps the site going. > > > We have two memcached servers and currently use them in such way: > > > -look in memcached1 for value > > > -if not found -> do a querry - write result in memcached1 and 2 > > > -if memcached1 is not available -> look in memcached1 for values > > > I know that's not the proper way to do this, but at the time we didn't > > > have that much spare servers. > > > Now the problem is this. > > > During normal operation, these are memcached stats: > > > STAT pid 7995 > > > STAT uptime 4984 > > > STAT time 1254317043 > > > STAT version 1.4.1 > > > STAT pointer_size 32 > > > STAT rusage_user 334.184196 > > > STAT rusage_system 915.517820 > > > STAT curr_connections 529 > > > STAT total_connections 4098258 > > > STAT connection_structures 727 > > > STAT cmd_get 55994089 > > > STAT cmd_set 2103657 > > > STAT cmd_flush 0 > > > STAT get_hits 51099680 > > > STAT get_misses 4894409 > > > STAT delete_misses 147084 > > > STAT delete_hits 916142 > > > STAT incr_misses 0 > > > STAT incr_hits 0 > > > STAT decr_misses 0 > > > STAT decr_hits 0 > > > STAT cas_misses 0 > > > STAT cas_hits 0 > > > STAT cas_badval 0 > > > STAT bytes_read 2259781347 > > > STAT bytes_written 30945451617 > > > STAT limit_maxbytes 0 > > > STAT accepting_conns 1 > > > STAT listen_disabled_num 0 > > > STAT threads 9 > > > STAT conn_yields 0 > > > STAT bytes 235845560 > > > STAT curr_items 175108 > > > STAT total_items 2103657 > > > STAT evictions 0 > > > > However from time to time (once a week, sometimes twice a day), the > > > connections to the server jump to about 3000 and stay that way, the DB > > > gets flooded with connections, those who can - connect and do queries, > > > the rest stay in TIME_WAIT. During normal operation the DB (mysql) has > > > about 30 active connections at a given time and about 100 TIME_WAIT- > > > ing. > > > When the memcached problem appears, the connections to the DB jump to > > > 2000 (the max amount) and the ones waiting are over 40K. > > > Memcached restart (both servers) solves the problem - gradually in > > > 10minutes time, the most frequent queries are cached and everything > > > gets quiet again. I made a script that sets a key with value the > > > current time and expire time 50 seconds, 60 seconds later checks if > > > that key is still there. 999 times out of 1000 the key has expired but > > > sometimes I get a message from the script that the key is still there. > > > I also tried setting 10000 unique keys with expire time 50 seconds, 60 > > > seconds later get all of them, well all had expired as expected, but I > > > still have not run that while the memcached servers are in "strange- > > > behaviour-mode". > > > Do you have any ideas/suggestions - at least how to diagnose the > > > problem, or how to reproduce it and what could be the problem? > > > Currently I have a cronjob that restarts one server every odd hour and > > > the other every even hour, but that's not pretty. > > > We are in the process of setting up around 10 memcached servers in a > > > standard server array, hopefully that will solve the problem. > > > > Thanks in advance and sorry for the long post, > > > Cheers
