Under loaded conditions (3 machines doing httperf separately), I did a separate wget on the side, and am attaching the TCPDUMP of that request. As you can see, there is a delay in the middle where varnish didn't respond immediately. If thread/hit-rate conditions are optimal, this delay should be minimal I thought.
Any help would be appreciated. -T On Fri, Mar 4, 2011 at 2:30 PM, Tejaswi Nadahalli <[email protected]>wrote: > On Fri, Mar 4, 2011 at 2:25 PM, Caunter, Stefan <[email protected]>wrote: > >> There’s no health check in the backend. Not sure what that does with a one >> hour grace. I set a short grace with >> >> >> >> if (req.backend.healthy) { >> >> set req.grace = 60s; >> >> } else { >> >> set req.grace = 4h; >> >> } >> > > I am still to add health-checks, directors, etc. Will add them soon. But > those make sense if the cache-primed performance is good. In my test, I am > requesting URLs who I know are already in the cache. Varnishstat also shows > that - there are no cache misses at all. > > >> >> >> You also don’t appear to select a backend in recv. >> > > The default backend seems to be getting picked up automatically. > > -T > > >> >> >> Stefan Caunter >> >> Operations >> >> Torstar Digital >> >> m: (416) 561-4871 >> >> >> >> >> >> *From:* [email protected] [mailto: >> [email protected]] *On Behalf Of *Tejaswi Nadahalli >> *Sent:* March-04-11 1:23 PM >> >> *To:* [email protected] >> *Subject:* Re: Under Load: Server Unavailable/Connection Dropped/Delayed >> Reponse >> >> >> >> On Fri, Mar 4, 2011 at 9:43 AM, Caunter, Stefan <[email protected]> >> wrote: >> >> >> >> What does something like firebug show when you request during the load >> test? The delay may be anything from DNS to the ec2 network. >> >> >> The DNS requests are getting resolved super quick. I am unable to see any >> other network issues with EC2. I have a similar machine in the same data >> center running nginx which is doing similar loads, but with no caching >> requirement, and it's running fine. >> >> In my first post, I forgot to attach my VCL, which is a bit too minimal. >> Am I missing something obvious? >> >> ------ >> backend default0 { >> .host = "10.202.30.39"; >> .port = "8000"; >> } >> >> sub vcl_recv { >> unset req.http.Cookie; >> set req.grace = 3600s; >> set req.url = regsub(req.url, "&refurl=.*&t=.*&c=.*&r=.*", ""); >> } >> >> sub vcl_deliver { >> if (obj.hits > 0) { >> set resp.http.X-Cache = "HIT"; >> } else { >> set resp.http.X-Cache = "MISS"; >> } >> } >> ------------------------- >> >> Could there be some kind of TCP packet pileup that I am missing? >> >> -T >> >> >> >> >> Stefan Caunter >> >> Operations >> >> Torstar Digital >> >> m: (416) 561-4871 >> >> >> >> >> >> *From:* [email protected] [mailto: >> [email protected]] *On Behalf Of *Tejaswi Nadahalli >> *Sent:* March-04-11 1:09 AM >> *To:* [email protected] >> *Subject:* Under Load: Server Unavailable/Connection Dropped/Delayed >> Reponse >> >> >> >> Hi Everyone, >> >> I am seeing a situation similar to : >> >> >> http://www.varnish-cache.org/lists/pipermail/varnish-misc/2011-January/005351.html(Connections >> Dropped Under Load) >> >> http://www.varnish-cache.org/lists/pipermail/varnish-misc/2010-December/005258.html(Hanging >> Connections) >> >> I have httperf loading a varnish cache with never-expire content. While >> the load is on, other browser/wget requests to the varnish server get >> delayed to 10+ seconds. Any ideas what could be happening? ssh doesn't seem >> to be impacted. So, is it some kind of thread problem? >> >> In production, I see a similar situation with around 1000 req/second load. >> >> >> I am running varnishd with the following command line options (as per >> http://kristianlyng.wordpress.com/2009/10/19/high-end-varnish-tuning/): >> >> sudo varnishd -f /etc/varnish/default.vcl -s malloc,5G -T 127.0.0.1:2000-a >> 0.0.0.0:80 -p thread_pools=8 -p thread_pool_min=100 -p >> thread_pool_max=5000 -p thread_pool_add_delay=2 -p cli_timeout=25 -p >> session_linger=100 -p lru_interval=20 -t 31536000 >> >> I am on Ubuntu Lucid 64 bit Amazon EC2 C1.XLarge with 8 processing units. >> >> My network sysctl parameters are tuned according to: >> http://varnish-cache.org/trac/wiki/Performance >> fs.file-max = 360000 >> net.ipv4.ip_local_port_range = 1024 65536 >> net.core.rmem_max = 16777216 >> net.core.wmem_max = 16777216 >> net.ipv4.tcp_rmem = 4096 87380 16777216 >> net.ipv4.tcp_wmem = 4096 65536 16777216 >> net.ipv4.tcp_fin_timeout = 3 >> net.core.netdev_max_backlog = 30000 >> net.ipv4.tcp_no_metrics_save = 1 >> net.core.somaxconn = 262144 >> net.ipv4.tcp_syncookies = 0 >> net.ipv4.tcp_max_orphans = 262144 >> net.ipv4.tcp_max_syn_backlog = 262144 >> net.ipv4.tcp_synack_retries = 2 >> net.ipv4.tcp_syn_retries = 2 >> >> >> Any help would be greatly appreciated >> >> >> > >
20:15:46.896200 IP 208.64.111.126.7544 > 10.202.30.39.80: Flags [S], seq 975218147, win 5840, options [mss 1460,sackOK,TS val 239507633 ecr 0,nop,wscale 6], length 0 20:15:46.896220 IP 10.202.30.39.80 > 208.64.111.126.7544: Flags [S.], seq 2642556500, ack 975218148, win 5792, options [mss 1460,sackOK,TS val 267323553 ecr 239507633,nop,wscale 9], length 0 20:15:46.932874 IP 208.64.111.126.7544 > 10.202.30.39.80: Flags [.], ack 1, win 92, options [nop,nop,TS val 239507639 ecr 267323553], length 0 20:15:46.932900 IP 208.64.111.126.7544 > 10.202.30.39.80: Flags [P.], seq 1:341, ack 1, win 92, options [nop,nop,TS val 239507639 ecr 267323553], length 340 20:15:46.933404 IP 10.202.30.39.80 > 208.64.111.126.7544: Flags [.], ack 341, win 14, options [nop,nop,TS val 267323556 ecr 239507639], length 0 20:16:07.129730 IP 10.202.30.39.80 > 208.64.111.126.7544: Flags [.], seq 1:2897, ack 341, win 14, options [nop,nop,TS val 267325576 ecr 239507639], length 2896 20:16:07.129752 IP 10.202.30.39.80 > 208.64.111.126.7544: Flags [.], seq 2897:4345, ack 341, win 14, options [nop,nop,TS val 267325576 ecr 239507639], length 1448 20:16:07.138422 IP 208.64.111.126.7544 > 10.202.30.39.80: Flags [.], ack 1449, win 137, options [nop,nop,TS val 239512697 ecr 267325576], length 0 20:16:07.138439 IP 10.202.30.39.80 > 208.64.111.126.7544: Flags [.], seq 4345:5793, ack 341, win 14, options [nop,nop,TS val 267325577 ecr 239512697], length 1448 20:16:07.138446 IP 10.202.30.39.80 > 208.64.111.126.7544: Flags [P.], seq 5793:5998, ack 341, win 14, options [nop,nop,TS val 267325577 ecr 239512697], length 205 20:16:07.138450 IP 208.64.111.126.7544 > 10.202.30.39.80: Flags [.], ack 2897, win 182, options [nop,nop,TS val 239512697 ecr 267325576], length 0 20:16:07.138456 IP 208.64.111.126.7544 > 10.202.30.39.80: Flags [.], ack 4345, win 227, options [nop,nop,TS val 239512697 ecr 267325576], length 0 20:16:07.148340 IP 208.64.111.126.7544 > 10.202.30.39.80: Flags [.], ack 5793, win 273, options [nop,nop,TS val 239512699 ecr 267325577], length 0 20:16:07.148350 IP 208.64.111.126.7544 > 10.202.30.39.80: Flags [.], ack 5998, win 318, options [nop,nop,TS val 239512699 ecr 267325577], length 0 20:16:07.148353 IP 208.64.111.126.7544 > 10.202.30.39.80: Flags [F.], seq 341, ack 5998, win 318, options [nop,nop,TS val 239512699 ecr 267325577], length 0 20:16:07.148441 IP 10.202.30.39.80 > 208.64.111.126.7544: Flags [F.], seq 5998, ack 342, win 14, options [nop,nop,TS val 267325578 ecr 239512699], length 0 20:16:07.156951 IP 208.64.111.126.7544 > 10.202.30.39.80: Flags [.], ack 5999, win 318, options [nop,nop,TS val 239512702 ecr 267325578], length 0
_______________________________________________ varnish-misc mailing list [email protected] http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
