Hi folks, for the past few days I've been puzzled by this problem - Varnish have refused to cache even static files. While it was fine before, and will cache even dynamic HTML files (with proper cache headers, of course)
The problem begun when my client reported that his website was constantly going up & down. When I checked, it was under some sort of DDoS attack. This is not news, because his website was pretty much under 24x7 syn flood attack. However, this attack is now able to bypass HAproxy & Varnish - and hit Apache directly, right where it hurts most. To cut the long story short - basically now Varnish refuses to cache almost everything. The X-Cacheable header that I enabled contains the dreaded "NO:Not Cacheable" status. And I've not been able to find out why. This is where I stuck, if I can find out the cause, the I'd be able to rectify it. Anyway, the DDoS attack may not be related at all to the current problem (varnish not caching), I included the story just for the sake of completeness. Let's proceed to some facts that I've gathered : Same GIF file - but different sizes (note the "Received" column) : http://minus.com/mbawzSZUxJ#3 The URLs with port 8181 are direct requests to Varnish, bypassing HAproxy. HTTP Headers produced by Varnish : http://minus.com/mbawzSZUxJ#4 HTTP Headers produced by HAproxy : http://minus.com/mbawzSZUxJ#2 List of cookies sent by browser & received from Varnish : http://minus.com/mbawzSZUxJ#1 (stripped clean by varnish, basically) Please find the result of varnishstat -1 & content of /etc/varnish/default.vcl attached to the end of this email. Varnish is running with the following parameters : $ ps aux|grep varnish myuser 3327 0.0 0.0 3324 796 pts/6 S+ 04:36 0:00 grep varnish root 8441 0.0 1.0 86024 83532 pts/2 SL+ Jan12 0:01 /usr/sbin/varnishd -P /var/run/varnishd.pid -a 0.0.0.0:8181 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 180 -w 2,1000,30 -s malloc,2G -d Hopefully someone will be able to point / give me a hint to the right direction. Thanks, Harry =================== $ telnet localhost 6082 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. 200 199 ----------------------------- Varnish Cache CLI 1.0 ----------------------------- Linux,2.6.32-25-generic-pae,i686,-smalloc,-hcritbit Type 'help' for command list. Type 'quit' to close CLI session. stats 200 1978 16564302 Client connections accepted 16514647 Client requests received 2378064 Cache hits 988836 Cache misses 592750 Backend conn. success 462880 Backend conn. failures 13096234 Backend conn. reuses 63305 Backend conn. was closed 13160380 Backend conn. recycles 82 Fetch head 9937334 Fetch with Length 1856 Fetch chunked 245 Fetch wanted close 66 Fetch failed 26436 N struct sess_mem 26091 N struct sess 18773 N struct object 18862 N struct objectcore 6827 N struct objecthead 3 N struct vbe_conn 107 N worker threads 26496 N worker threads created 144188 N overflowed work requests 1 N backends 932755 N expired objects 1851315 N LRU moved objects 12437230 Objects sent with write 16564302 Total Sessions 16514647 Total Requests 870 Total pipe 13146877 Total pass 13630039 Total fetch 6464791313 Total header bytes 122615308544 Total body bytes 16564289 Session Closed 11 Session Linger 17 Session herd 1364697125 SHM records 93867958 SHM writes 2574 SHM flushes due to overflow 446883 SHM MTX contention 540 SHM cycles through buffer 10815252 SMA allocator requests 37554 SMA outstanding allocations 45497812 SMA outstanding bytes 114575871831 SMA bytes allocated 114530374019 SMA bytes free 505674 SMS allocator requests 211877406 SMS bytes allocated 211877406 SMS bytes freed 13688813 Backend requests made 1 N vcl total 1 N vcl available 1 N total active purges 1 N new purges added 3365986 HCB Lookups without lock 122890 HCB Lookups with lock 122890 HCB Inserts 99485 Client uptime 16426 Backend conn. retry 3690588 Fetch no body (304) =================== default.vcl =============== $ cat /etc/varnish/default.vcl # Default backend definition. Set this to point to your content # server. backend default { .host = "127.0.0.1"; .port = "81"; } sub vcl_deliver { if (obj.hits > 0) { set resp.http.X-Cache = "HIT"; } else { set resp.http.X-Cache = "MISS"; } } # Below is a commented-out copy of the default VCL logic. If you # redefine any of these subroutines, the built-in logic will be # appended to your code. # sub vcl_recv { if (req.backend.healthy) { set req.grace = 180s; } else { set req.grace = 1h; } } sub vcl_fetch { set beresp.grace = 1h; # Varnish determined the object was not cacheable if (!beresp.cacheable) { set beresp.http.X-Cacheable = "NO:Not Cacheable"; # You don't wish to cache content for logged in users } elsif (req.http.Cookie ~ "(UserID|_session)") { set beresp.http.X-Cacheable = "NO:Got Session"; return(pass); # You are respecting the Cache-Control=private header from the backend } elsif (beresp.http.Cache-Control ~ "private") { set beresp.http.X-Cacheable = "NO:Cache-Control=private"; return(pass); # You are extending the lifetime of the object artificially } elsif (beresp.ttl < 1s) { set beresp.ttl = 5s; set beresp.grace = 5s; set beresp.http.X-Cacheable = "YES:FORCED"; # Varnish determined the object was cacheable } else { set beresp.http.X-Cacheable = "YES"; } if (req.url ~ "\.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|pdf|txt|tar|wav|bmp|rtf|js|flv|swf|html|htm)$") { unset beresp.http.set-cookie; set beresp.ttl = 24h; } return(deliver); # if (!beresp.cacheable) { # return (pass); # } # if (beresp.http.Set-Cookie) { # return (pass); # } # return (deliver); } _______________________________________________ varnish-misc mailing list [email protected] https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
