Hi, We are running Varnish 2.0.6 on two machines (dual quad core CPU, kernel 2.6.18-194.3.1.el5) through a load balancer. Both Varnishes are connected to load-balanced (fast responding) back-ends, each handling on average 500 requests/s (97+% hits).
Each once and a while (in the range from once an hour to once a week) we see an instant worker thread pile-up with overflowed and sometimes dropped work requests. During these short periods (about 15min) everything returns to normal. The pile-ups occur at non regular intervals at both quiet and busy times. Pile-ups occur on both machines at the same time. During pile-ups Varnish responses are extremely slow and transfer rates drop disrupting our services. Furthermore we sometimes see the following in /var/log/messages: Dec 19 22:40:04 cache1 kernel: possible SYN flooding on port 80. Sending cookies. Dec 19 22:41:05 cache1 kernel: possible SYN flooding on port 80. Sending cookies. We've been looking closely at internal (back-ends) and external (network) factors but so far haven't found a cause of the problems described above. Anyone has a clue what could be the cause of this problem? Or recognizes this issue? Thanks for your time, Dennis --- Varnish: Varnish config (storage: malloc 10GB): accept_fd_holdoff 50 [ms] acceptor default (epoll, poll) auto_restart on [bool] backend_http11 on [bool] between_bytes_timeout 60.000000 [s] cache_vbe_conns off [bool] cc_command "exec cc -fpic -shared -Wl,-x -o %o %s" cli_buffer 8192 [bytes] cli_timeout 5 [seconds] client_http11 off [bool] clock_skew 10 [s] connect_timeout 0.400000 [s] default_grace 10 default_ttl 120 [seconds] diag_bitmap 0x0 [bitmap] err_ttl 0 [seconds] esi_syntax 0 [bitmap] fetch_chunksize 128 [kilobytes] first_byte_timeout 60.000000 [s] group varnish (103) listen_address :80 listen_depth 1024 [connections] log_hashstring off [bool] log_local_address off [bool] lru_interval 2 [seconds] max_esi_includes 5 [includes] max_restarts 4 [restarts] obj_workspace 8192 [bytes] overflow_max 100 [%] ping_interval 3 [seconds] pipe_timeout 60 [seconds] prefer_ipv6 off [bool] purge_dups on [bool] purge_hash on [bool] rush_exponent 3 [requests per request] send_timeout 600 [seconds] sess_timeout 5 [seconds] sess_workspace 16384 [bytes] session_linger 50 [ms] session_max 100000 [sessions] shm_reclen 255 [bytes] shm_workspace 8192 [bytes] srcaddr_hash 1049 [buckets] srcaddr_ttl 0 [seconds] thread_pool_add_delay 20 [milliseconds] thread_pool_add_threshold 2 [requests] thread_pool_fail_delay 200 [milliseconds] thread_pool_max 1000 [threads] thread_pool_min 15 [threads] thread_pool_purge_delay 1000 [milliseconds] thread_pool_stack unlimited [bytes] thread_pool_timeout 300 [seconds] thread_pools 8 [pools] user varnish (101) vcl_trace off [bool] _______________________________________________ varnish-misc mailing list [email protected] http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
