too many clients
Helo I have got a problem using varnish infront of webserwer cluster running arroung 20K req/s; when i tried to put varnish(4 loadbalanced machines 8CPU cores each 32GB ram) infront of this farm it serves OK for few seconds but then aper in syslog such messages: kernel: TCP: drop open request from IP/port and varnish stops serving I thing it is related with number of clients (which was around 30K when the problem apeard) and rate of request from each of this clients ( arround 1 request each 5 seconds). I have tried tunning kernel TCP stack ... are there any recomandation on varnish seting for many clients? Vaclav Bilek my params: accept_fd_holdoff 50 [ms] acceptor default (epoll, poll) auto_restart on [bool] backend_http11 on [bool] between_bytes_timeout 60.00 [s] cache_vbe_connsoff [bool] cc_command exec cc -fpic -shared -Wl,-x -o %o %s cli_buffer 8192 [bytes] cli_timeout5 [seconds] client_http11 off [bool] clock_skew 10 [s] connect_timeout0.40 [s] default_grace 10 default_ttl60 [seconds] diag_bitmap0x0 [bitmap] err_ttl0 [seconds] esi_syntax 0 [bitmap] fetch_chunksize128 [kilobytes] first_byte_timeout 60.00 [s] group nogroup (65534) listen_address 0.0.0.0:80 listen_depth 10240 [connections] log_hashstring off [bool] log_local_address off [bool] lru_interval 2 [seconds] max_esi_includes 5 [includes] max_restarts 4 [restarts] obj_workspace 8192 [bytes] overflow_max 100 [%] ping_interval 3 [seconds] pipe_timeout 60 [seconds] prefer_ipv6off [bool] purge_dups off [bool] purge_hash on [bool] rush_exponent 3 [requests per request] send_timeout 600 [seconds] sess_timeout 1 [seconds] sess_workspace 16384 [bytes] session_linger 0 [ms] shm_reclen 255 [bytes] shm_workspace 8192 [bytes] srcaddr_hash 1049 [buckets] srcaddr_ttl30 [seconds] thread_pool_add_delay 20 [milliseconds] thread_pool_add_threshold 2 [requests] thread_pool_fail_delay 200 [milliseconds] thread_pool_max2048 [threads] thread_pool_min500 [threads] thread_pool_purge_delay1000 [milliseconds] thread_pool_timeout10 [seconds] thread_pools 4 [pools] user nobody (65534) vcl_trace off [bool] stats: 7514 Client connections accepted 35229 Client requests received 27328 Cache hits 57 Cache hits for pass 5396 Cache misses 7901 Backend connections success 0 Backend connections not attempted 0 Backend connections too many 0 Backend connections failures 0 Backend connections reuses 0 Backend connections recycles 0 Backend connections unused 5203 N struct srcaddr 3758 N active struct srcaddr 4520 N struct sess_mem 4298 N struct sess 5433 N struct object 1623 N struct objecthead 6923 N struct smf 0 N small free smf 32 N large free smf 2 N struct vbe_conn 19 N struct bereq 2000 N worker threads 2000 N worker threads created 0 N worker threads not created 0 N worker threads limited 0 N queued work requests 4 N overflowed work requests 0 N dropped work requests 1 N backends 34 N expired objects 0 N LRU nuked objects 0 N LRU saved objects 3239 N LRU moved objects 0 N objects on deathrow 0 HTTP header overflows 0 Objects sent with sendfile 8973 Objects sent with write 0 Objects overflowing workspace 7514 Total Sessions 35225 Total Requests 0 Total pipe 2505 Total pass 7899 Total fetch 3497006 Total header bytes 32744211 Total body bytes 631 Session Closed 0 Session Pipeline 0 Session Read Ahead 0 Session Linger 34676 Session herd 1399198 SHM records 96468 SHM writes 2 SHM flushes due to overflow 10 SHM MTX contention 0 SHM cycles through buffer 10926 allocator requests 6891 outstanding allocations 57233408 bytes allocated 29953720320 bytes free 0 SMA allocator requests 0 SMA outstanding allocations 0 SMA outstanding bytes 0
Re: Varnish memory consumption issues?
On Sat, Aug 29, 2009 at 12:09:15AM -0400, pub crawler wrote: Hello, new to Varnish. We have been running Varnish for about 5 days now. So far, excellent product. We have a potential issue and I haven't seen anything like this before. We just restarted Varnish - we have a 1GB cache file on disk. When I run top, I see Varnish is using : 10m RES 153g VIRT I'll leave a more detail explanation to PHK or someone more experienced with the deep magics of a VM, but the 10m is the real memory consumption which I'm sure top/free can confirm. Though 153G seem very high for VIRT. What I usually see is VIRT = 2x cache size, or thereabouts, but that might not be conclusive. Do you care to paste your startup arguments and parameters? -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgp3jBn43z5mb.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish memory consumption issues?
Thanks Kristian for the explanation. I posted this, and I believe it likely was our issue - perhaps not starting Cherokee with proper parameters included. Our configuration is pretty barbones default per se. What is troubling is 153GB is more resources than this machine has RAM + disk combined. Stumped as to how Varnish or the OS would allocate all this memory when the resources not there. It's worth noting that we changed our commandline to spawn Varnish and Varnish seems to be more reasonable with memory now: 5775 nobody20 0 4363m 169m 162m S1 8.6 1:00.03 varnishd 4GB is the size of the cache file we setup for Cherokee so VIRT is right on consumption wise. RES memory at 169m is quite fine / reasonable also. These numbers seem to what others experience right? -Paul On Wed, Sep 2, 2009 at 9:25 AM, Kristian Lyngstolkrist...@redpill-linpro.com wrote: On Sat, Aug 29, 2009 at 12:09:15AM -0400, pub crawler wrote: Hello, new to Varnish. We have been running Varnish for about 5 days now. So far, excellent product. We have a potential issue and I haven't seen anything like this before. We just restarted Varnish - we have a 1GB cache file on disk. When I run top, I see Varnish is using : 10m RES 153g VIRT I'll leave a more detail explanation to PHK or someone more experienced with the deep magics of a VM, but the 10m is the real memory consumption which I'm sure top/free can confirm. Though 153G seem very high for VIRT. What I usually see is VIRT = 2x cache size, or thereabouts, but that might not be conclusive. Do you care to paste your startup arguments and parameters? -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Child Died
I just started my first instance of varnish in production. Within 12 hours, there were alerts from our monitoring system that Varnish was taking 90% of the cpu. Right after that, I find these messages in /var/log/messages, several times over a 2 minute period: varnishd[12461]: Child (20086) not responding to ping, killing it. The child restarted, and the stats and cache all disappeared. This is a machine with 8 gigs of ram and a pair of slightly older quad core xeons. The storage method is file with a 50 gig limit. At its peak, the machine is serving around 40 requests a second, about 5000k a second. The configs are the defaults. What should my first steps be to troubleshoot this? Is there a likely culprit? ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Flushing buffer during page delivery
Hello -- We have our apache children flush the buffer when generating a page after emitting the JS/CSS includes, after emitting the page framework, etc. However, despite that we're flushing the buffer to Varnish, it doesn't seem to deliver any of the data to the user until the page is complete. Although the page generation time is minimal, flushing the buffer early lets the user's browser start pulling dependencies while apache spends another 100ms or so delivering the rest of the page. Is there any way to get Varnish to flush the buffer regularly or accomplish this in some way? Thanks! Daniel Baker ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
