Hi Harry, Is HAProxy in front of your Varnish servers or between varnish and Apache?
You could use HAProxy to mitigate the attack, if you know the pattern, I can help on this point. cheers On Thu, Jan 12, 2012 at 10:37 PM, Harry Sufehmi <[email protected]> wrote: > Hi folks, for the past few days I've been puzzled by this problem - > Varnish have refused to cache even static files. While it was fine > before, and will cache even dynamic HTML files (with proper cache > headers, of course) > > The problem begun when my client reported that his website was > constantly going up & down. When I checked, it was under some sort of > DDoS attack. This is not news, because his website was pretty much > under 24x7 syn flood attack. > However, this attack is now able to bypass HAproxy & Varnish - and hit > Apache directly, right where it hurts most. > > To cut the long story short - basically now Varnish refuses to cache > almost everything. > The X-Cacheable header that I enabled contains the dreaded "NO:Not > Cacheable" status. And I've not been able to find out why. > > This is where I stuck, if I can find out the cause, the I'd be able to > rectify it. > > Anyway, the DDoS attack may not be related at all to the current > problem (varnish not caching), I included the story just for the sake > of completeness. > > Let's proceed to some facts that I've gathered : > > Same GIF file - but different sizes (note the "Received" column) : > http://minus.com/mbawzSZUxJ#3 > > The URLs with port 8181 are direct requests to Varnish, bypassing HAproxy. > > HTTP Headers produced by Varnish : http://minus.com/mbawzSZUxJ#4 > > HTTP Headers produced by HAproxy : http://minus.com/mbawzSZUxJ#2 > > List of cookies sent by browser & received from Varnish : > http://minus.com/mbawzSZUxJ#1 > (stripped clean by varnish, basically) > > Please find the result of varnishstat -1 & content of > /etc/varnish/default.vcl attached to the end of this email. > > Varnish is running with the following parameters : > > $ ps aux|grep varnish > myuser 3327 0.0 0.0 3324 796 pts/6 S+ 04:36 0:00 grep varnish > root 8441 0.0 1.0 86024 83532 pts/2 SL+ Jan12 0:01 > /usr/sbin/varnishd -P /var/run/varnishd.pid -a 0.0.0.0:8181 -f > /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 180 -w 2,1000,30 -s > malloc,2G -d > > > Hopefully someone will be able to point / give me a hint to the right > direction. > > > Thanks, > Harry > > =================== > > $ telnet localhost 6082 > Trying 127.0.0.1... > Connected to localhost. > Escape character is '^]'. > 200 199 > ----------------------------- > Varnish Cache CLI 1.0 > ----------------------------- > Linux,2.6.32-25-generic-pae,i686,-smalloc,-hcritbit > > Type 'help' for command list. > Type 'quit' to close CLI session. > > stats > 200 1978 > 16564302 Client connections accepted > 16514647 Client requests received > 2378064 Cache hits > 988836 Cache misses > 592750 Backend conn. success > 462880 Backend conn. failures > 13096234 Backend conn. reuses > 63305 Backend conn. was closed > 13160380 Backend conn. recycles > 82 Fetch head > 9937334 Fetch with Length > 1856 Fetch chunked > 245 Fetch wanted close > 66 Fetch failed > 26436 N struct sess_mem > 26091 N struct sess > 18773 N struct object > 18862 N struct objectcore > 6827 N struct objecthead > 3 N struct vbe_conn > 107 N worker threads > 26496 N worker threads created > 144188 N overflowed work requests > 1 N backends > 932755 N expired objects > 1851315 N LRU moved objects > 12437230 Objects sent with write > 16564302 Total Sessions > 16514647 Total Requests > 870 Total pipe > 13146877 Total pass > 13630039 Total fetch > 6464791313 Total header bytes > 122615308544 Total body bytes > 16564289 Session Closed > 11 Session Linger > 17 Session herd > 1364697125 SHM records > 93867958 SHM writes > 2574 SHM flushes due to overflow > 446883 SHM MTX contention > 540 SHM cycles through buffer > 10815252 SMA allocator requests > 37554 SMA outstanding allocations > 45497812 SMA outstanding bytes > 114575871831 SMA bytes allocated > 114530374019 SMA bytes free > 505674 SMS allocator requests > 211877406 SMS bytes allocated > 211877406 SMS bytes freed > 13688813 Backend requests made > 1 N vcl total > 1 N vcl available > 1 N total active purges > 1 N new purges added > 3365986 HCB Lookups without lock > 122890 HCB Lookups with lock > 122890 HCB Inserts > 99485 Client uptime > 16426 Backend conn. retry > 3690588 Fetch no body (304) > > =================== default.vcl =============== > > $ cat /etc/varnish/default.vcl > > # Default backend definition. Set this to point to your content > # server. > backend default { > .host = "127.0.0.1"; > .port = "81"; > } > > > sub vcl_deliver { > if (obj.hits > 0) { > set resp.http.X-Cache = "HIT"; > } else { > set resp.http.X-Cache = "MISS"; > } > } > > # Below is a commented-out copy of the default VCL logic. If you > # redefine any of these subroutines, the built-in logic will be > # appended to your code. > # > sub vcl_recv { > > if (req.backend.healthy) { > set req.grace = 180s; > } else { > set req.grace = 1h; > } > } > > > > sub vcl_fetch { > > set beresp.grace = 1h; > > # Varnish determined the object was not cacheable > if (!beresp.cacheable) { > set beresp.http.X-Cacheable = "NO:Not Cacheable"; > > # You don't wish to cache content for logged in users > } elsif (req.http.Cookie ~ "(UserID|_session)") { > set beresp.http.X-Cacheable = "NO:Got Session"; > return(pass); > > # You are respecting the Cache-Control=private header from the backend > } elsif (beresp.http.Cache-Control ~ "private") { > set beresp.http.X-Cacheable = "NO:Cache-Control=private"; > return(pass); > > # You are extending the lifetime of the object artificially > } elsif (beresp.ttl < 1s) { > set beresp.ttl = 5s; > set beresp.grace = 5s; > set beresp.http.X-Cacheable = "YES:FORCED"; > > # Varnish determined the object was cacheable > } else { > set beresp.http.X-Cacheable = "YES"; > } > > if (req.url ~ > "\.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|pdf|txt|tar|wav|bmp|rtf|js|flv|swf|html|htm)$") > { > unset beresp.http.set-cookie; > set beresp.ttl = 24h; > } > > return(deliver); > > # if (!beresp.cacheable) { > # return (pass); > # } > # if (beresp.http.Set-Cookie) { > # return (pass); > # } > # return (deliver); > } > > _______________________________________________ > varnish-misc mailing list > [email protected] > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc _______________________________________________ varnish-misc mailing list [email protected] https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
