Hi guys, So sometimes are backend is really slow in returning a response. So slow that it looks like it is causing varnish to timeout before the backend does and so varnish seems to eventually give up and returns a 503.
Ther are 4 curl requests * the first 2 timeout and give a 503 * the 3rd one is a hit miss but gives a 200 * the 4th one is a cache hit and gives a 200 https://gist.github.com/0452c374ee21dbba138d Here's the varnishlog filtered for client requests http://gist.github.com/279254e0f2452814bf46 Here's varnishstat https://gist.github.com/219534d51a503b546070 Here's are current startup run time parameters options right now: https://gist.github.com/219534d51a503b546070 So, Im pretty certain what is happening is that our backend takes too long and varnish times out. How can I set the time out higher... is it sess_timeout? Thanks guys, Tung On Wed, Mar 25, 2009 at 10:37 AM, Tung Nguyen <[email protected]>wrote: > Kristian, thank you. > > Im glad to hear that most defaults are good. > > Yup, the default thread_pool_min = 1 seems kinda funny. We'll set it to at > least 100. > > I didnt even know about the cli_timeout and will set it to a base line of > 10 seconds to test. > > Im wondering how you are testing, Im using ab, apachebench, to see how > things behave with -c 10 -n 1000, on the varnished pages. > > > > > > Here's more specific questions more run time parameters. The general > question I have is what to look for during testing, should I be looking at > your varnishstat and are the most important things to look for in that > output. > > > Our varnish stack will look like this: > > LB -> Varnish x 2 -> Nginx x 6 -> Mongrel x 60 > > Some questions about how best to decide how to configure them best to > configure the run time parameters. > > -p obj_workspace=4096 > Cant find obj_workspace in the man page but found it in the twitter email > post > http://projects.linpro.no/pipermail/varnish-dev/2009-February/000968.html > > Is obj_workspace how much space preallocated to be used for the obj that > gets returned from the backend? So, if my nginx backend returns a web page > that is over 4MB than -p obj_workspace is not enough, would that crash > varnish, or log the error somewhere. > > -p sess_workspace=262144 > Same deal here with the man page and twitter post. > What is the sess_workspace? > > http_workspace > How does sess_workspace and obj_workspace relate to http_workspace? > If we use obj_workspace=4096 and sess_workspace=262144, does the default > http_workspace=8192 make sense? > > > -p lru_interval=60 > Shows up on the twitter post again, but no man notes yet. Whats the > default for this? > > -p sess_timeout=10 \ > Default for this is 5. If the requests from the backend takes longer than > 5 seconds, what happens? Sometimes we have really slow response from the > backend.. > > -p shm_workspace=32768 \ > Is this the same as setting the command line flag -l shmlogsize. The > default is 80MB. So dont know twitter did both setting it to less.. > > -p thread_pools=4 \ > -p thread_pool_min=100 \ > thread_pool_max > The defaults are 1,1,1000 respectively. Im wondering how best to determine > this or just leave as default. > > > > > > ###################################################################### > # output of varnishstat, what is best to look at here? > ###################################################################### > 0+07:11:24 > Hitrate ratio: 4 4 > 4 > Hitrate avg: nan nan nan > > 400 0.00 0.02 Client connections accepted > 400 0.00 0.02 Client requests received > 1 0.00 0.00 Cache hits > 5 0.00 0.00 Cache misses > 399 0.00 0.02 Backend connections success > 399 0.00 0.02 Backend connections failures > 1 0.00 0.00 Backend connections reuses > 5 0.00 0.00 Backend connections recycles > 6 . . N struct srcaddr > 21 . . N struct sess_mem > 1 . . N struct sess > 1 . . N struct object > 1 . . N struct objecthead > 3 . . N struct smf > 1 . . N small free smf > 1 . . N large free smf > 1 . . N struct vbe_conn > 2 . . N struct bereq > 10 . . N worker threads > 23 0.00 0.00 N worker threads created > 76 0.00 0.00 N overflowed work requests > 2 . . N backends > 5 . . N expired objects > 6 0.00 0.00 Objects sent with write > 400 0.00 0.02 Total Sessions > 400 0.00 0.02 Total Requests > 394 0.00 0.02 Total pipe > 5 0.00 0.00 Total fetch > 352131 0.00 13.60 Total header bytes > 1745 0.00 0.07 Total body bytes > 397 0.00 0.02 Session Closed > 3 0.00 0.00 Session herd > 33185 2.00 1.28 SHM records > 19306 2.00 0.75 SHM writes > 32 0.00 0.00 SHM MTX contention > 11 0.00 0.00 allocator requests > 1 . . outstanding allocations > 8192 . . bytes allocated > 6612459520 . . bytes free > 5 0.00 0.00 Backend requests made > 1 0.00 0.00 N vcl total > 1 0.00 0.00 N vcl available > 1 . . N total active purges > 1 0.00 0.00 N new purges added > > > > > > > > Any response is appreciated. > > Thanks, > Tung > > > > > > > > > > > On Tue, Mar 24, 2009 at 12:20 AM, Kristian Lyngstol < > [email protected]> wrote: > >> On Mon, Mar 23, 2009 at 05:58:58PM -0700, Tung Nguyen wrote: >> > Hi guys, >> > So, Im reading over an archive email thread about twitters >> configuration. >> > >> > >> http://projects.linpro.no/pipermail/varnish-dev/2009-February/000968.html >> > >> > It looks like they had to adjust a lot of parameters... and Im not >> finding >> > all the parameters definitions in the varnishd man pages. Im wondering >> if >> > for most cases running varnish with the defaults is fine? >> > >> > Any caveats here which run time parameters should I focus on. >> >> You can mostly run it with the defaults, yes. This depends on what sort of >> usage you have though. >> >> One notable exception is that I strongly recommend that you bring >> thread_pool_min up to a decent level (reflecting how many users you >> actually have. Numbers in the hundreds is normal). You'll also want to >> adjust the cache size to your system, but that goes without saying. >> >> If you expect extremely high load, you might have to increase cli_timeout >> too. I've run tests where even setting it to 15 seconds is insufficient >> and >> causes childs to be killed off. Though for most production sites, I'd >> guess >> 5 seconds could work and 10 seconds would definitely work. >> >> -- >> Kristian Lyngstøl >> Redpill Linpro AS >> Tlf: +47 21544179 >> Mob: +47 99014497 >> > > > > -- > Tung Nguyen, Lead Developer > Bleacher Report, The Open Source Sports Network > (510) 928-0475 > -- Tung Nguyen, Lead Developer Bleacher Report, The Open Source Sports Network (510) 928-0475
_______________________________________________ varnish-misc mailing list [email protected] http://projects.linpro.no/mailman/listinfo/varnish-misc
