Hank,
Thanks for your help on this. I am working with Amanda on this project.
1 - I installed PCRE from source and built with your original make
arguments.
2 - uname -a 2.6.18-164.el5xen
3 - we have spun up a new version of Red Hat and these are our polling
options now:
Available polling systems :
sepoll : pref=400, test result OK
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 4 (4 usable), will use sepoll.
I am not quite sure where to go from here to debug the problem
further. If anyone would like to make a few dollars consulting for
us, I would be happy to SSH access to our servers to allow some
hands-on troubleshooting.
Thanks!
Geoff
On Tue, Mar 30, 2010 at 8:06 PM, Hank A. Paulson <
[email protected]> wrote:
> And pcre static is a separate package for Fedora/RedHat.
>
> For RH/Fedora:
>
> yum install pcre-devel pcre
>
> make TARGET=linux26 ARCH==x86-64 USE_REGPARM=1 USE_PCRE=1
>
> then you can do:
> ./haproxy -vv
>
> to make sure you have several poll options, just having select seems
> strange.
>
> Also, nbproc should be 1 I can push 70+ million hits/day and 100Mbps on a
> VM with nbproc = 1, so you should be ok with nbproc = 1 for non-extreme
> traffic.
> [Also, Willy will freak out if he sees you using nbproc > 1 :) j/k]
>
> If there are ip_conntrack problems, you will usually see some log messages
> in the system log, so when your traffic goes up generally ip_conntrack is
> the first bottleneck (the ip_conntrack people don't like to admit that,
> but...).
>
> what is your uname -a? Are you running a recent kernel?
>
> Also, one common cause of pauses like this is dns/reverse dns lookups, so
> be sure those are turned off on your web server configs. I don't think this
> is the main problem in your case, but it is something to note. And for
> tcpdump, you usually want to turn that off tcpdump -nn... on linux:
> -n Don’t convert host addresses to names. This can be used to avoid DNS
> lookups.
> -nn Don’t convert protocol and port numbers etc. to names either.
>
> You might want to try running the http part of your service in haproxy http
> mode and use the related http close options.
>
>
> On 3/30/10 8:57 AM, Malte Geierhos wrote:
>
>> Hi there,
>>
>> either install a libpcre-dev package so you've got the header files - or
>> forget about that in the first shot,
>> and just make TARGET=linux26
>>
>> regards,
>> Malte
>>
>> I tried upgrading with the switches you recommended, but am getting
>>> errors during the make. I've installed pcre. Any suggestions? Thanks!
>>>
>>> [r...@rs-globallb1 haproxy-1.4.2]# make TARGET=linux26 USE_REGPARM=1
>>> USE_STATIC_PCRE=1 USE_PCRE=1
>>> gcc -Iinclude -Iebtree -Wall -O2 -g -DTPROXY -DCONFIG_HAP_CRYPT
>>> -DENABLE_POLL -DENABLE_EPOLL -DENABLE_SEPOLL -DNETFILTER
>>> -DUSE_GETSOCKNAME -DCONFIG_REGPARM=3 -DUSE_PCRE -I/usr/local/include
>>> -DCONFIG_HAPROXY_VERSION=\"1.4.2\" -DCONFIG_HAPROXY_DATE=\"2010/03/17\" \
>>> -DBUILD_TARGET='"linux26"' \
>>> -DBUILD_ARCH='""' \
>>> -DBUILD_CPU='"generic"' \
>>> -DBUILD_CC='"gcc"' \
>>> -DBUILD_CFLAGS='"-O2 -g"' \
>>> -DBUILD_OPTIONS='"USE_REGPARM=1 USE_PCRE=1 USE_STATIC_PCRE=1"' \
>>> -c -o src/haproxy.o src/haproxy.c
>>> In file included from include/types/proxy.h:33,
>>> from include/common/cfgparse.h:29,
>>> from src/haproxy.c:54:
>>> include/common/regex.h:28:18: error: pcre.h: No such file or directory
>>> include/common/regex.h:29:23: error: pcreposix.h: No such file or
>>> directory
>>> In file included from include/types/proxy.h:33,
>>> from include/common/cfgparse.h:29,
>>> from src/haproxy.c:54:
>>> include/common/regex.h:45: error: expected â:â, â,â, â;â, â}â or
>>> â__attribute__â before â*â token
>>> include/common/regex.h:51: error: expected â=â, â,â, â;â, âasmâ or
>>> â__attribute__â before âpmatchâ
>>> include/common/regex.h:53: warning: type defaults to âintâ in
>>> declaration of âregmatch_tâ
>>> include/common/regex.h:53: error: expected â;â, â,â or â)â before â*â
>>> token
>>> include/common/regex.h:55: warning: type defaults to âintâ in
>>> declaration of âregex_tâ
>>> include/common/regex.h:55: error: expected â;â, â,â or â)â before â*â
>>> token
>>> In file included from include/types/proxy.h:38,
>>> from include/common/cfgparse.h:29,
>>> from src/haproxy.c:54:
>>> include/types/acl.h:218: error: expected specifier-qualifier-list
>>> before âregex_tâ
>>> src/haproxy.c: In function âdeinitâ:
>>> src/haproxy.c:767: error: âstruct hdr_expâ has no member named âpregâ
>>> src/haproxy.c:768: warning: implicit declaration of function âregfreeâ
>>> src/haproxy.c:768: error: âregex_tâ undeclared (first use in this
>>> function)
>>> src/haproxy.c:768: error: (Each undeclared identifier is reported only
>>> once
>>> src/haproxy.c:768: error: for each function it appears in.)
>>> src/haproxy.c:768: error: expected expression before â)â token
>>> src/haproxy.c:769: error: expected expression before â)â token
>>> src/haproxy.c:772: error: âstruct hdr_expâ has no member named âreplaceâ
>>> src/haproxy.c:772: error: âstruct hdr_expâ has no member named âactionâ
>>> src/haproxy.c:773: error: âstruct hdr_expâ has no member named âreplaceâ
>>> src/haproxy.c:780: error: âstruct hdr_expâ has no member named âpregâ
>>> src/haproxy.c:781: error: expected expression before â)â token
>>> src/haproxy.c:782: error: expected expression before â)â token
>>> src/haproxy.c:785: error: âstruct hdr_expâ has no member named âreplaceâ
>>> src/haproxy.c:785: error: âstruct hdr_expâ has no member named âactionâ
>>> src/haproxy.c:786: error: âstruct hdr_expâ has no member named âreplaceâ
>>>
>>>
>>> On Tue, Mar 30, 2010 at 11:22 AM, Malte Geierhos <[email protected]
>>> <mailto:[email protected]>> wrote:
>>>
>>> Hi Amanda,
>>>
>>> try - using 1.4.2 - the 1.4.1 has issues with chunked encoding
>>> under some circumstances ...
>>> maybe you just hit that issue.
>>> On the other hand ... there's something weird with your build, too...
>>> You should at least have more polling systems -> mine looks like
>>> this :
>>>
>>> Available polling systems :
>>> sepoll : pref=400, test result OK
>>> epoll : pref=300, test result OK
>>> poll : pref=200, test result OK
>>>
>>> select : pref=150, test result OK
>>> Total: 4 (4 usable), will use sepoll.
>>> Using sepoll() as the polling mechanism.
>>>
>>> Try building like this :
>>> make TARGET=linux26 USE_REGPARM=1 USE_STATIC_PCRE=1 USE_PCRE=1
>>>
>>> And with 1.4.2 ...
>>>
>>> nf_conntrack on a loadbalancer can have a huge impact if you're
>>> serving lots of requests - so
>>> on a production system i'd try to disable this ( you can define
>>> acl's in haproxy for disabling certain clients and so on)
>>> for the rest like ssh use tcpwrapper or let it listen on a
>>> dedicated ip.
>>>
>>> kind regards,
>>> Malte
>>>
>>>
>>> Am 30.03.10 16:57, schrieb Amanda Machutta:
>>>
>>>> Just some more debugging data -- this is debuggng from
>>>> haproxy....I marked where the pause happens here as well:
>>>>
>>>> Available polling systems :
>>>> select : pref=150, test result OK
>>>> Total: 1 (1 usable), will use select.
>>>> Using select() as the polling mechanism.
>>>> 00000000:services.accept(0005)=0006 from [24.99.160.41:52838
>>>> <http://24.99.160.41:52838>]
>>>>
>>>> 00000001:http_proxy.accept(0003)=0008 from [24.99.160.41:52839
>>>> <http://24.99.160.41:52839>]
>>>>
>>>> 00000001:http_proxy.clireq[0008:ffff]: GET /intelliqueue/agentui/
>>>> HTTP/1.1
>>>> 00000001:http_proxy.clihdr[0008:ffff]: Host: www.cfpolitical.com
>>>> <http://www.cfpolitical.com>
>>>>
>>>> 00000001:http_proxy.clihdr[0008:ffff]: User-Agent: Mozilla/5.0
>>>> (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.2) Gecko/20100316
>>>> Firefox/3.6.2 (.NET CLR 3.5.30729)
>>>> 00000001:http_proxy.clihdr[0008:ffff]: Accept:
>>>> text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
>>>> 00000001:http_proxy.clihdr[0008:ffff]: Accept-Language:
>>>> en-us,en;q=0.5
>>>> 00000001:http_proxy.clihdr[0008:ffff]: Accept-Encoding: gzip,deflate
>>>> 00000001:http_proxy.clihdr[0008:ffff]: Accept-Charset:
>>>> ISO-8859-1,utf-8;q=0.7,*;q=0.7
>>>> 00000001:http_proxy.clihdr[0008:ffff]: Keep-Alive: 115
>>>> 00000001:http_proxy.clihdr[0008:ffff]: Connection: keep-alive
>>>> 00000001:http_proxy.clihdr[0008:ffff]: Cookie: CFID=408;
>>>> CFTOKEN=12213899; JSESSIONID=b6302e0e4a8d377c4e6e
>>>> 00000001:http_proxy.srvrep[0008:0009]: HTTP/1.1 200 OK
>>>> 00000001:http_proxy.srvhdr[0008:0009]: Connection: close
>>>> 00000001:http_proxy.srvhdr[0008:0009]: Date: Tue, 30 Mar 2010
>>>> 14:55:39 GMT
>>>> 00000001:http_proxy.srvhdr[0008:0009]: Server: Microsoft-IIS/6.0
>>>> 00000001:http_proxy.srvhdr[0008:0009]: X-Powered-By: ASP.NET
>>>> <http://ASP.NET>
>>>>
>>>> 00000001:http_proxy.srvhdr[0008:0009]: Set-Cookie:
>>>> CFAUTHORIZATION_IntelliQueue=;expires=Mon, 30-Mar-2009 14:55:39
>>>> GMT;path=/
>>>> 00000001:http_proxy.srvhdr[0008:0009]: Content-Type: text/html;
>>>> charset=UTF-8
>>>> 00000000:services.srvcls[0006:0007]
>>>> 00000000:services.clicls[0006:0007]
>>>> 00000000:services.closed[0006:0007]
>>>>
>>>> <!--- 16 second pause --->
>>>>
>>>> 00000001:http_proxy.srvcls[0008:0009]
>>>> 00000001:http_proxy.clicls[0008:0009]
>>>> 00000001:http_proxy.closed[0008:0009]
>>>> 00000002:http_proxy.accept(0003)=0006 from [24.99.160.41:52843
>>>> <http://24.99.160.41:52843>]
>>>>
>>>> 00000002:http_proxy.clireq[0006:ffff]: GET
>>>> /intelliqueue/agentui/config.cfm HTTP/1.1
>>>> 00000002:http_proxy.clihdr[0006:ffff]: Host: www.cfpolitical.com
>>>> <http://www.cfpolitical.com>
>>>>
>>>> 00000002:http_proxy.clihdr[0006:ffff]: User-Agent: Mozilla/5.0
>>>> (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.2) Gecko/20100316
>>>> Firefox/3.6.2 (.NET CLR 3.5.30729)
>>>> 00000002:http_proxy.clihdr[0006:ffff]: Accept:
>>>> text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
>>>> 00000002:http_proxy.clihdr[0006:ffff]: Accept-Language:
>>>> en-us,en;q=0.5
>>>> 00000002:http_proxy.clihdr[0006:ffff]: Accept-Encoding: gzip,deflate
>>>> 00000002:http_proxy.clihdr[0006:ffff]: Accept-Charset:
>>>> ISO-8859-1,utf-8;q=0.7,*;q=0.7
>>>> 00000002:http_proxy.clihdr[0006:ffff]: Keep-Alive: 115
>>>> 00000002:http_proxy.clihdr[0006:ffff]: Connection: keep-alive
>>>> 00000002:http_proxy.clihdr[0006:ffff]: Cookie: CFID=408;
>>>> CFTOKEN=12213899; JSESSIONID=b6302e0e4a8d377c4e6e
>>>> 00000002:http_proxy.srvrep[0006:0007]: HTTP/1.1 200 OK
>>>> 00000002:http_proxy.srvhdr[0006:0007]: Connection: close
>>>> 00000002:http_proxy.srvhdr[0006:0007]: Date: Tue, 30 Mar 2010
>>>> 14:55:56 GMT
>>>> 00000002:http_proxy.srvhdr[0006:0007]: Server: Microsoft-IIS/6.0
>>>> 00000002:http_proxy.srvhdr[0006:0007]: X-Powered-By: ASP.NET
>>>> <http://ASP.NET>
>>>>
>>>> 00000002:http_proxy.srvhdr[0006:0007]: Set-Cookie:
>>>> CFAUTHORIZATION_IntelliQueue=;expires=Mon, 30-Mar-2009 14:55:56
>>>> GMT;path=/
>>>> 00000002:http_proxy.srvhdr[0006:0007]: Content-Type: text/xml
>>>> 00000002:http_proxy.srvcls[0006:0007]
>>>> 00000002:http_proxy.clicls[0006:0007]
>>>> 00000002:http_proxy.closed[0006:0007]
>>>> 00000003:http_proxy.accept(0003)=0006 from [24.99.160.41:52844
>>>> <http://24.99.160.41:52844>]
>>>>
>>>> 00000003:http_proxy.clireq[0006:ffff]: GET
>>>> /intelliqueue/manage/get_logo.cfm?ui=TRUE HTTP/1.1
>>>> 00000003:http_proxy.clihdr[0006:ffff]: Host: www.cfpolitical.com
>>>> <http://www.cfpolitical.com>
>>>>
>>>> 00000003:http_proxy.clihdr[0006:ffff]: User-Agent: Mozilla/5.0
>>>> (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.2) Gecko/20100316
>>>> Firefox/3.6.2 (.NET CLR 3.5.30729)
>>>> 00000003:http_proxy.clihdr[0006:ffff]: Accept:
>>>> text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
>>>> 00000003:http_proxy.clihdr[0006:ffff]: Accept-Language:
>>>> en-us,en;q=0.5
>>>> 00000003:http_proxy.clihdr[0006:ffff]: Accept-Encoding: gzip,deflate
>>>> 00000003:http_proxy.clihdr[0006:ffff]: Accept-Charset:
>>>> ISO-8859-1,utf-8;q=0.7,*;q=0.7
>>>> 00000003:http_proxy.clihdr[0006:ffff]: Keep-Alive: 115
>>>> 00000003:http_proxy.clihdr[0006:ffff]: Connection: keep-alive
>>>> 00000003:http_proxy.clihdr[0006:ffff]: Cookie: CFID=408;
>>>> CFTOKEN=12213899; JSESSIONID=b6302e0e4a8d377c4e6e
>>>> 00000003:http_proxy.srvrep[0006:0007]: HTTP/1.1 200 OK
>>>> 00000003:http_proxy.srvhdr[0006:0007]: Connection: close
>>>> 00000003:http_proxy.srvhdr[0006:0007]: Date: Tue, 30 Mar 2010
>>>> 14:55:56 GMT
>>>> 00000003:http_proxy.srvhdr[0006:0007]: Server: Microsoft-IIS/6.0
>>>> 00000003:http_proxy.srvhdr[0006:0007]: X-Powered-By: ASP.NET
>>>> <http://ASP.NET>
>>>>
>>>> 00000003:http_proxy.srvhdr[0006:0007]: Content-Type: image/png
>>>> 00000003:http_proxy.srvcls[0006:0007]
>>>> 00000003:http_proxy.clicls[0006:0007]
>>>> 00000003:http_proxy.closed[0006:0007]
>>>> 00000004:services.accept(0005)=0006 from [24.99.160.41:52846
>>>> <http://24.99.160.41:52846>]
>>>>
>>>> 00000004:services.srvcls[0006:0007]
>>>> 00000004:services.clicls[0006:0007]
>>>> 00000004:services.closed[0006:0007]
>>>> 00000005:services.accept(0005)=0006 from [24.99.160.41:52847
>>>> <http://24.99.160.41:52847>]
>>>>
>>>
>>>
>>
>
