On Sat, Jan 16, 2010 at 3:55 PM, Michael Fischer <[email protected]> wrote: > This scheme seems very baroque. Why not just reduce the size of your caches > so you don't page-thrash and let Varnish's builtin LRU algorithm handle the > eviction?
Then I wont be able to cache nearly as much. I want to originate as much content as possible on the varnish servers ie. reduce backend fetches. There is no way I could fit any useful amount of my working set into a storage that could handle the evictions without spending an unreasonable amount of money (basically fit it in RAM.) -I'd love to be proven wrong though. As far as random reads go, the SSD's are really good; it's just the writes that kill me. Right now a mostly filled cache server with ~80-160GB allocated can maintain between 90-92% cache hit ratio at 400-500Mb/sec. When it fills up completely eviction cause the machine to keel over, parent can't ping the child, health checks fail -general badness. I'd like to let the eviction run under supervision (automated supervision) and augment the eviction such that it buys back a few hours not minutes. > --Michael > > On Sat, Jan 16, 2010 at 2:46 PM, David Birdsong <[email protected]> > wrote: >> >> I'm trying to hack my way around a push-button like lru nuking like >> feature. The short description of how I'm doing it follows, I'll >> explain why farther down. >> >> I have a job that watches sm_bfree / (sm_bfree + sm_balloc). Once >> storage file utilization is past some percentage(yet to be determined) >> I connect to upstream load balancers and slowly drain traffic away >> from varnish. >> >> Once traffic is off and I can beat the hell out of that box, it's time >> to free up some space. In the past this has been done with restarts. >> Upon restarts, the cache hit ratio is destroyed, but the box can keep >> up and rebuild the cache in a stable way. What I'd like to do is dump >> everything in the storage files that have a very low obj.hits. Lru >> nuking on the surface seems like the best thing to initiate, but it >> usually only kicks in pretty late and puts the machine into a state >> that is unstable while serving. While not serving, I don't know how >> to kick it off, furthermore I want it to run hard and free up lots >> more space than it usually does. >> >> ie. cache file is ~200GB, I'd like it to run until sm_free is like 50GB. >> >> My idea is to load balance as I've described above. Pull 50GB of >> trash files through the cache + enough to kick off lru, purge the >> trash files, monitor sm_bfree and once it's high enough instruct the >> upstream load balancers to start sending traffic gently for a warm up >> period. Rinse and repeat into infinity replacing the ssd storage >> drives as they fail. Is this crazy? Am I uninformed on a better way? >> >> Also, I've had to keep making my trash files smaller and smaller. I >> started with a 10 and 1G files which crashed varnish immediately, then >> reduced to 500MB files and successfully pulled 200 through - then >> crashed both my python interpreter (libcurl) and varnish: >> varnishd[2664]: Child (14772) Panic message: Assert error in >> STV_alloc(), stevedore.c line 183:#012 Condition((st) != NULL) not >> true.#012thread = (cache-worker)#012Backtrace:#012 0x421f95: >> pan_ic+85#012 0x4369e5: STV_alloc+125#012 0x41a1b6: >> FetchBody+496#012 0x4114dd: cnt_fetch+63d#012 0x412a3d: >> CNT_Session+35d#012 0x424273: wrk_do_cnt_sess+93#012 0x42362e: >> wrk_thread_real+26e#012 0x7f2cf51b83da: _end+7f2cf4b47c1a#012 >> 0x7f2cf4a862bd: _end+7f2cf4415afd#012sp = 0x7f2ced387008 {#012 fd = >> 58, id = 58, xid = 1454039386,#012 client = 127.0.0.1:7057,#012 step >> = STP_FETCH,#012 handling = deliver,#012 err_code = 200, err_reason >> = (null),#012 restarts = 0, esis = 0#012 ws = 0x7f2ced387078 { #012 >> id = "sess",#012 {s,f,r,e} = >> {0x7f2ced387800,+144,(nil),+4096},#012 },#012 http[req] = {#012 >> ws = 0x7f2ced387078[sess]#012 "GET",#012 >> "/lru.10.cache.buster.80.12994",#012 "HTTP/1.1",#012 >> "User-Agent: PycURL/7.18.2",#012 "Host: localhost:6081",#012 >> "Accept: */*",#012 },#012 worker = 0x7ef439f06390 {#012 ws = >> 0x7ef439f068f0 { #012 id = "wrk",#012 {s,f,r,e} = >> {0x7ef439f03350,+2143,(nil),+4096},#012 },#012 http[bereq] = >> {#012 ws = 0x7ef439f068f0[wrk]#012 "GET",#012 >> "/lru.10.cache.buster.80.12994",#012 "HTTP/1.1",#012 >> "User-Agent: PycURL/7.18.2",#012 "Host: localhost:6081",#012 >> "Accept: */*",#012 "X-Varnish: 1454039386",#012 >> "X-Forwarded-For: 127.0.0.1",#012 },#012 http[beresp] = {#012 >> ws = 0x7ef439f068f0[wrk]#012 "HTTP/1.1",#012 >> "200",#012 "OK",#012 "Server: nginx/0.7.64",#012 >> "Date: Sat, 16 Jan 2010 21:11:09 GMT",#012 "Content-Type: >> application/octet-stream",#012 "Content-Length: 524288000",#012 >> "Last-Modified: Sat, 16 Jan 2010 21:08:11 GMT",#012 >> "Connection: keep-alive",#012 "Accept-Ranges: bytes",#012 >> "X-Varnish-IP: 127.0.0.1",#012 "X-Varnish-Port: 6081",#012 >> },#012 },#012 >> >> Are big files bad? I expect that I'll have to close a pretty big gap >> normally given that my 4 storage files are 75GB each (SSD). I'd like >> to start this process before lru nuking happens on it's own while >> varnish is not unloaded by upstream load balancers. My guess based on >> loose recollection is that varnish will start lru nuking at 90% >> capacity. It may just prove not feasible given that I'll have to pull >> roughly 60GB through to achieve the goal....perhaps freeing up a >> smaller percentage would be acceptable too though. I'm still playing >> with this, but wanted to share my uber-hacky idea and let you guys >> tear it apart if it's a dumb idea. >> >> Why: >> Identifying the working set has been difficult. It's large, the long >> tail is very long. I've tried adaptive ttls to expire objects >> constantly that shouldn't be in cache: >> >> in vcl_fetch: set every new object to a 2hr ttl. >> in vcl_hit: if obt.hits == N ; then obj.ttl = 36 hours, where N is >> some number that is high enough to cache >> another permutation, update the vcl every 30 mins such that obj.ttl >> was set to expire exactly at the trough of traffic (2300 - 2350 PST) >> in vcl_hit: if obt.hits = N ; then obj.ttl = 12h or 10h, or 3h >> (depending on time of day) >> >> This just ended up affecting cache hit ratio such that it was never >> favorable and the box was just busier as it was constantly expiring >> objects over the day. Restarts were still better than this. >> >> Setup: >> >> 3 haproxy load balancer machines consistently hashing to 6 varnish >> instances. It's a prototype and will be scaled to a larger pool, so >> the impact of the downtime of a single varnish instance while it goes >> through a cache storage scrubbing is will be greatly reduced. >> _______________________________________________ >> varnish-misc mailing list >> [email protected] >> http://projects.linpro.no/mailman/listinfo/varnish-misc > > _______________________________________________ varnish-misc mailing list [email protected] http://projects.linpro.no/mailman/listinfo/varnish-misc
