Fair enough. We don't use keep-alive, so it hasn't been an issue. What are you guys using for load balancing?
best, cloude On Mon, Mar 2, 2009 at 1:45 PM, Artur Bergman <s...@crucially.net> wrote: > "Right now, HAProxy only supports the first mode (HTTP close) if it needs to > process the request. This means that for each request, there will be one TCP > connection. If keep-alive or pipelining are required, HAProxy will still > support them, but will only see the first request and the first response of > each transaction. While this is generally problematic with regards to logs, > content switching or filtering, it most often causes no problem for > persistence > with cookie insertion." > Really not fully supporting keep-alive! More like varnish pipe mode > Artur > On Mar 2, 2009, at 1:33 PM, Cloude Porteus wrote: > >> I believe TCP Keep-alive has been supported in HAProxy since version >> 1.2. We've been using 1.3.x for at least a year. >> >> -cloude >> >> On Mon, Mar 2, 2009 at 1:10 PM, Artur Bergman <s...@crucially.net> wrote: >>> >>> HAProxy doesn't do keep-alive, so it makes everything slower. >>> >>> Artur >>> >>> On Feb 27, 2009, at 9:02 PM, Cloude Porteus wrote: >>> >>>> John, >>>> Thanks so much for the info, that's a huge help for us!!! >>>> >>>> I love HAProxy and Willy has been awesome to us. We run everything >>>> through it, since it's really easy to monitor and also easy to debug >>>> where the lag is when something in the chain is not responding fast >>>> enough. It's been rock solid for us. >>>> >>>> The nice part for us is that we can use it as a content switcher to >>>> send all /xxx traffic or certain user-agent traffic to different >>>> backends. >>>> >>>> best, >>>> cloude >>>> >>>> On Fri, Feb 27, 2009 at 2:24 PM, John Adams <j...@twitter.com> wrote: >>>>> >>>>> cc'ing the varnish dev list for comments... >>>>> On Feb 27, 2009, at 1:33 PM, Cloude Porteus wrote: >>>>> >>>>> John, >>>>> Goodto hear from you. You must be slammed at Twitter. I'm happy to >>>>> hear that ESI is holding up for you. It's been in my backlog since you >>>>> mentioned it to me pre-Twitter. >>>>> >>>>> Any performance info would be great. >>>>> >>>>> >>>>> Any comments on our setup are welcome. You may also choose to call us >>>>> crazypants. Many, many thanks to Artur Bergman of Wikia for helping us >>>>> get >>>>> this configuration straightened out. >>>>> Right now, we're running varnish (on search) in a bit of a non-standard >>>>> way. >>>>> We plan to use it in the normal fashion (varnish to Internet, nothing >>>>> inbetween) on our API at some point. We're running version 2.0.2, no >>>>> patches. Cache hit rates range from 10% to 30%, or higher when a >>>>> real-time >>>>> event is flooding search. >>>>> 2.0.2 is quite stable for us, with the occasional child death here and >>>>> there >>>>> when we get massive headers coming in that flood sess_workspace. I hear >>>>> this >>>>> is fixed in 2.0.3, but haven't had time to try it yet. >>>>> We have a number of search boxes, and each search box has an apache >>>>> instance >>>>> on it, and varnish instance. We plan to merge the varnish instances at >>>>> some >>>>> point, but we use very low TTLs (Twitter is the real time web!) and >>>>> don't >>>>> see much of a savings by running less of them. >>>>> We do: >>>>> Apache --> Varnish --> Apache -> Mongrels >>>>> Apaches are using mod_proxy_balancer. The front end apache is there >>>>> because >>>>> we've long had a fear that Varnish would crash on us, which it did many >>>>> times prior to our figuring out the proper parameters for startup. We >>>>> have >>>>> two entries in that balancer. Either the request goes to varnish, or, >>>>> if >>>>> varnish bombs out, it goes directly to the mongrel. >>>>> We do this, because we need a load balancing algorithm that varnish >>>>> doesn't >>>>> support, called bybusiness. Without bybusiness, varnish tries to direct >>>>> requests to Mongrels that are busy, and requests end up in the listen >>>>> queue. >>>>> that adds ~100-150mS to load times, and that's no good for our desired >>>>> service times of 200-250mS (or less.) >>>>> We'd be so happy if someone put bybusiness into Varnish's backend load >>>>> balancing, but it's not there yet. >>>>> We also know that taking the extra hop through localhost costs us next >>>>> to >>>>> nothing in service time, so it's good to have Apache there incase we >>>>> need >>>>> to >>>>> yank out Varnish. In the future, we might get rid of Apache and use >>>>> HAProxy >>>>> (it's load balancing and backend monitoring is much richer than Apache, >>>>> and, >>>>> it has a beautiful HTTP interface to look at.) >>>>> Some variables and our decisions: >>>>> -p obj_workspace=4096 \ >>>>> -p sess_workspace=262144 \ >>>>> Absolutely vital! Varnish does not allocate enough space by default >>>>> for >>>>> headers, regexps on cookies, and otherwise. It was increased in 2.0.3, >>>>> but >>>>> really, not increased enough. Without this we were panicing every 20-30 >>>>> requests and overflowing the sess hash. >>>>> -p listen_depth=8192 \ >>>>> 8192 is probably excessive for now. If we're queuing 8k conns, >>>>> something >>>>> is >>>>> really broke! >>>>> -p log_hashstring=off \ >>>>> Who cares about this - we don't need it. >>>>> -p lru_interval=60 \ >>>>> We have many small objects in the search cache. Run LRU more often. >>>>> -p sess_timeout=10 \ >>>>> If you keep session data around for too long, you waste memory. >>>>> -p shm_workspace=32768 \ >>>>> Give us a bit more room in shm >>>>> -p ping_interval=1 \ >>>>> Frequent pings in case the child dies on us. >>>>> -p thread_pools=4 \ >>>>> -p thread_pool_min=100 \ >>>>> This must match up with VARNISH_MIN_THREADS. We use four pools, (pools >>>>> * >>>>> thread_pool_min == VARNISH_MIN_THREADS) >>>>> -p srcaddr_ttl=0 \ >>>>> Disable the (effectively unused) per source-IP statistics >>>>> -p esi_syntax=1 >>>>> Disable ESI syntax verification so we can use it to process JSON >>>>> requests. >>>>> If you have more than 2.1M objects, you should also add: >>>>> # -h classic,250007 = recommeded value for 2.1M objects >>>>> # number should be 1/10 expected working set. >>>>> >>>>> In our VCL, we have a few fancy tricks that we use. We label the cache >>>>> server and cache hit/miss rate in vcl_deliver with this code: >>>>> Top of VCL: >>>>> C{ >>>>> #include <stdio.h> >>>>> #include <unistd.h> >>>>> char myhostname[255] = ""; >>>>> >>>>> }C >>>>> vcl_deliver: >>>>> C{ >>>>> VRT_SetHdr(sp, HDR_RESP, "\014X-Cache-Svr:", myhostname, >>>>> vrt_magic_string_end); >>>>> }C >>>>> /* mark hit/miss on the request */ >>>>> if (obj.hits > 0) { >>>>> set resp.http.X-Cache = "HIT"; >>>>> set resp.http.X-Cache-Hits = obj.hits; >>>>> } else { >>>>> set resp.http.X-Cache = "MISS"; >>>>> } >>>>> >>>>> vcl_recv: >>>>> C{ >>>>> if (myhostname[0] == '\0') { >>>>> /* only get hostname once - restart required if hostname changes */ >>>>> gethostname(myhostname, 255); >>>>> } >>>>> }C >>>>> >>>>> Portions of /etc/sysconfig/varnish follow... >>>>> # The minimum number of worker threads to start >>>>> VARNISH_MIN_THREADS=400 >>>>> # The Maximum number of worker threads to start >>>>> VARNISH_MAX_THREADS=1000 >>>>> # Idle timeout for worker threads >>>>> VARNISH_THREAD_TIMEOUT=60 >>>>> # Cache file location >>>>> VARNISH_STORAGE_FILE=/var/lib/varnish/varnish_storage.bin >>>>> # Cache file size: in bytes, optionally using k / M / G / T suffix, >>>>> # or in percentage of available disk space using the % suffix. >>>>> VARNISH_STORAGE_SIZE="8G" >>>>> # >>>>> # Backend storage specification >>>>> VARNISH_STORAGE="malloc,${VARNISH_STORAGE_SIZE}" >>>>> # Default TTL used when the backend does not specify one >>>>> VARNISH_TTL=5 >>>>> # the working directory >>>>> DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \ >>>>> -f ${VARNISH_VCL_CONF} \ >>>>> -T >>>>> ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \ >>>>> -t ${VARNISH_TTL} \ >>>>> -n ${VARNISH_WORKDIR} \ >>>>> -w >>>>> ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT} >>>>> \ >>>>> -u varnish -g varnish \ >>>>> -p obj_workspace=4096 \ >>>>> -p sess_workspace=262144 \ >>>>> -p listen_depth=8192 \ >>>>> -p log_hashstring=off \ >>>>> -p lru_interval=60 \ >>>>> -p sess_timeout=10 \ >>>>> -p shm_workspace=32768 \ >>>>> -p ping_interval=1 \ >>>>> -p thread_pools=4 \ >>>>> -p thread_pool_min=100 \ >>>>> -p srcaddr_ttl=0 \ >>>>> -p esi_syntax=1 \ >>>>> -s ${VARNISH_STORAGE}" >>>>> >>>>> --- >>>>> John Adams >>>>> Twitter Operations >>>>> j...@twitter.com >>>>> http://twitter.com/netik >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> VP of Product Development >>>> Instructables.com >>>> >>>> http://www.instructables.com/member/lebowski >>>> _______________________________________________ >>>> varnish-dev mailing list >>>> varnish-dev@projects.linpro.no >>>> http://projects.linpro.no/mailman/listinfo/varnish-dev >>> >>> >> >> >> >> -- >> VP of Product Development >> Instructables.com >> >> http://www.instructables.com/member/lebowski > > -- VP of Product Development Instructables.com http://www.instructables.com/member/lebowski _______________________________________________ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev