CPU 100% when waiting for the client timeout

[baiyang]

Affect verions: at least 1.5.15 and 1.6.2

Here is the related part in my configuration:
timeout client  15m  # 客户端响应超时
timeout client-fin  10s  # 对客户端连接完成 TCP 4 次挥手超时
timeout connect 5s   # HAProxy 向后端 Server 建立连接时的等候超时
timeout server  15m  # 服务器端响应超时
timeout server-fin  5s   # 对后端 Server 连接完成 TCP 4 次挥手超时
timeout http-keep-alive 5m   # http keep-alive 
连接空闲（等待下一请求到达）保持超时（对前后端均有效，影响 http-reuse 选项）
timeout http-request15s  # http 请求 *头部* 接收超时（对前后端均有效）

And I found after upgraded to 1.6.2 and 1.5.15, my haproxy server offen using 
100% CPU. For example, here is the CPU usage graph of today:

I trace with pidstat (per 30 seconds) and haproxy debug level logging, And I 
found this record in the log:
Nov 19 08:35:12 WD-G0-SRP1 haproxy[28267]: 58.246.225.90:24989 
[19/Nov/2015:08:19:39.265] https-in~ g0n0/n0 724/0/0/1/933585 200 45430 - - 
CD-- 13/12/2/2/0 0/0 "GET /lib/libbaiy/charcode.mapping/chs.json HTTP/1.1"
Nov 19 14:53:51 WD-G0-SRP1 haproxy[28267]: 46.4.123.172:20214 
[19/Nov/2015:14:38:12.435] http-in WebSvr/ws0 0/0/1/2/939161 200 41488 - - CD-- 
52/21/0/0/0 0/0 "GET /app/wderp.apk HTTP/1.0"

These are the only two records show 15 minutes timeout today, and they are both 
at the and of the cpu usage spike. I believe this is not a coincidence. May be 
something is broken in the newest version of haproxy here?

--

   Best Regards
  BaiYang
  baiy...@gmail.com
  http://baiy.cn
 < END OF EMAIL >  
 

Re: Owncloud through Haproxy makes upload not possible

[Piotr Kubaj]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256



On 11/18/15 19:45, Bryan Talbot wrote:
> Hard to guess what the issue is but haproxy logging would probably
> help. However, the logging configuration is a bit of a mess as it's
> configured to be "all on" and "all off" at the same time. There are
> also other configuration oddities.

Well, I'm actually quite new to Haproxy, so I just followed some
random tutorial for domain proxy. It worked (I've got a few websites
on it), so I've just assumed it's ok.

> 4096 bit DH params will be pretty slow to handshake. Maybe that's
> okay in your circumstance though since you seem to be using this
> for a personal use and not expecting a high connection rate. You
> also have a 8 kbit RSA self signed certificate and using 256 bit
> ciphers which increase TLS overhead.
I want it to be secure, and I don't want to touch my settings for
quite a while so I just took the strongest algorithms there are, and
2x recommended values for things like private key, or DH params. The
hardware is pretty powerful and I've already checked that I don't have
a huge load.


> Is that where the logging socket is on FreeBSD now? I haven't used 
> FreeBSD in quite a while.
Yes, the default is /var/run/log.

> maxconn 4096 user daemon group daemon daemon
> 
> defaults modehttp option  httplog option  dontlognull option
> forwardfor option  http-server-close option  httpclose option
> tcplog option  dontlog-normal
> 
> 
> You have both tcp logging and http logging enabled at the same
> time. In addition, you also have all logging disabled with
> "dontlognull" and "dontlog-normal". If all your proxies are HTTP
> like you've shown, just enable httplog and remove the tcplog
> option. When troubleshooting, enable logging at least normal
> connections.
> 
> You also do not want to use both httpclose and http-server-close
> since they conflict. Remove option httpclose.
> 
> Timeouts are also missing and you should be getting warnings about
> that too.
> 
> [WARNING] 321/103152 (87887) : config : missing timeouts for proxy 
> 'https-in'. | While not properly invalid, you will certainly
> encounter various problems | with such a configuration. To fix
> this, please ensure that all following | timeouts are set to a
> non-zero value: 'client', 'connect', 'server'.
Well, it seems that the person that wrote the tutorial I used doesn't
know much about Haproxy either...

> frontend http-in bind 192.168.11.3:80  
> reqadd X-Forwarded-Proto:\ http redirect scheme https code 301 if
> !{ ssl_fc }
> 
> 

> 
> 
> 
> 
> 
> frontend https-in option httplog option forwardfor option
> http-server-close option httpclose
> 
> 
> Don't need to repeat all of the above since it should be set in
> defaults above (if set properly). With no "log global" you won't
> get any logs anyway and are probably seeing a warning when haproxy
> checks the config or starts.
> 
> [WARNING] 321/103109 (87884) : config : log format ignored for
> proxy 'https-in' since it has no log address.
> 
> 
> 
> 
> 
> 
> rspadd Public-Key-Pins:\ 
> pin-sha256="1Pw5h93NOsPw6j/vaTYl5VvW9cmtuZXtNP3cVz10hKo=";\ 
> max-age=15768000;\ includeSubDomains
> 
> 
> AFAIK, HPKP is only somewhat supported by only the most recent
> browser releases. I believe that it's also ignored by them for
> certificates which are self-signed or signed by a CA that is not in
> the browsers system-defined CA set. Probably doesn't cause your
> issue but who knows -- it is still experimental.
> 
> The "http-response set-header" supported in haproxy 1.5 and later
> is more powerful and easier to read than the old reqadd and rspadd
> features.
> 
> 
> 
> 
> bind 192.168.11.3:443  ssl crt 
> /usr/local/etc/haproxy.pem ciphers AES256+EECDH:AES256+EDH
> force-tlsv12 no-sslv3
> 
> 
> Don't need to repeat these options that are already set globally.
> 
> 
> 
> 
> 
> backend 10amd64 server node1 192.168.11.3:81
>  cookie A check
> 
> 
> 
> Setting sticky cookies and not using them is probably harmless but 
> what's the point?
> 
> -Bryan
> 


Debug logs from uploading a file:
00c1:https-in.clicls[0007:]
00c1:https-in.closed[0007:]
00c2:https-in.accept(0006)=0007 from [xxx:59266]
00c2:https-in.clireq[0007:]: POST
/index.php/apps/files/ajax/upload.php HTTP/1.1
00c2:https-in.clihdr[0007:]: Host: owncloud.anongoth.pl
00c2:https-in.clihdr[0007:]: User-Agent: Mozilla/5.0
(Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0
00c2:https-in.clihdr[0007:]: Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
00c2:https-in.clihdr[0007:]: Accept-Language: en-US, en
00c2:https-in.clihdr[0007:]: Accept-Encoding: gzip, deflate
00c2:https-in.clihdr[0007:]: DNT: 1
00c2:https-in.clihdr[0007:]: requesttoken:
e6185c74050ccda163a7f6d9299933a12b9fae66e46b97d1eed750c9ecd71dd2|C4wwTFE

Re: Echo server in Lua

[Willy Tarreau]

On Thu, Nov 19, 2015 at 07:00:32AM +, Thrawn wrote:
> Hmm...I haven't tuned anything (this is just my workstation, not a server).
> /proc/sys/net/core/somaxconn 128
> /proc/sys/net/ipv4/tcp_max_backlog doesn't exist
> 
>  /proc/sys/net/ipv4/tcp_max_syn_backlog512

Yep sorry it's this one.

> I guess those are pretty low for trying to thrash an echo server...

Yes they're low even for 16k/s.

> any recommendations?

Start by multiplying both of them by 4. If nothing improves it doesn't
come from there.

Willy

Re: appsession replacement in 1.6

[Aleksandar Lazic]

Hi Sylvain & Willy.

Am 19-11-2015 19:41, schrieb Willy Tarreau:

Hi Sylvain,

On Thu, Nov 19, 2015 at 05:04:40PM +0100, Sylvain Faivre wrote:

>What Aleks meant is that you don't need http-buffer-request as it's
>only used to process POST data which isn't your case.

Oh OK. Well, I did not talk about http-buffer-request in the 
beginning,

so I thought Aleks recommended to use it.
I can see that it is not useful in my case, so I removed it again.


I think he wanted to be on the safest side before knowing well all the
details of your setup. Now that it works, time for cleanup has come :-)


You got me Willy ;-)

For future reference, here is my latest config. It has been running 
for

a few hours on our staging servers without a problem :

backend front
[stick-table for another purpose]
[other stuff]
# sticky session on JSESSIONID
stick on urlp(jsessionid) table front_jsessionid
stick on urlp(jsessionid,;) table front_jsessionid
stick on cookie(JSESSIONID) table front_jsessionid
stick store-response cookie(JSESSIONID) table front_jsessionid

backend front_jsessionid
stick-table type string len 24 size 10m expire 1h peers prod


Thanks that's much appreciated. I wanted to work on a mini-doc to
explain how to replace appsession with stick tables but unfortunately I
failed to find the time to do it before the release. So I appreciate
that you share the result of your findings, it will definitely help
other users.

Thanks to you and Aleks for this :-)


Also thanks from me ;-)

Cheers
Aleks

Re: Owncloud through Haproxy makes upload not possible

[Piotr Kubaj]

On 11/19/2015 17:01, Janusz Dziemidowicz wrote:
> 2015-11-19 15:45 GMT+01:00 Piotr Kubaj :
>> Now, about RSA vs ECDSA. I simply don't trust ECDSA. There are quite a
>> lot of questions about constants used by ECDSA, which seem to be
>> chosen quite arbitrarily by its creator, which happens to be NSA.
>> These questions of course remain unanswered. Even respected scientists
>> like Schneier say that RSA should be used instead (see
>> https://www.schneier.com/blog/archives/2013/09/the_nsa_is_brea.html#c167
>> 5929
> 
> But ECDSA itself does not contain any constants (see
> https://en.wikipedia.org/wiki/Elliptic_Curve_Digital_Signature_Algorithm).
> Yes, you have to choose domain parameters and most commonly used are
> NIST ones. But you can also use brainpool curves, which specifically
> avoid using any arbitrary constants (see
> http://www.ecc-brainpool.org/download/Domain-parameters.pdf) and they
> are even defined for TLS (https://tools.ietf.org/html/rfc7027) and
> apparently supported by latest OpenSSL. Unfortunately not by anything
> else.
> OK, anyway that's your preference, I'm not going to argue about ECDSA or not;)
> 
>> ). When I'm done setting my HTTP(S) services, I'll simply limit
>> incoming connections connections on my firewall so DDOS'ing won't be
>> possible, unless you DDOS my firewall :)
> 
> I've never said anything about DDoS. In such setup there is no need
> for distributed DoS. The CPU usage of RSA 8192 is so high that a
> single shell script running on a single attack machine can kill any
> server.
> If you are willing to limit your connection rate on a firewall to a
> few per second, then fine;)
> 
> As for your problem. Now that it seems like SSL problem, can you just
> try with RSA 4096 or 2048? RSA 8192 is really not much tested in most
> code, so maybe the problem is in fact related.
> 
Unfortunately, accessing my HTTPS services by only OpenSSL is out of the
question. Besides, I use LibreSSL and am not sure it supports it, since
OpenBSD people got rid of quite a lot of unnecessary code.

So I can only choose ECDSA or RSA.

I don't think limiting my connections is a bad idea vs choosing weaker
RSA. As I said before, I actually expect only a few connections at once.

I've generated RSA 2048 cert with:
openssl req -x509 -newkey rsa:2048 -keyout haproxy.pem -out haproxy.pem
-days 3650 -nodes

That is, I didn't use any non-default options, such as SHA512.
Unfortunately, it doesn't yield any result. I'm now considering
switching to SSL Pass-through, and configuring HTTPS in each of my WWW
servers, it may be much quicker considering how long I've been getting
Haproxy to work.



signature.asc
Description: OpenPGP digital signature

Re: RE: CPU 100% when waiting for the client timeout

[Willy Tarreau]

Hello,

On Fri, Nov 20, 2015 at 12:09:03AM +0800, baiyang wrote:
> > Which release did you use previously that worked fine?
> I'm using 1.5.14 before upgraded to 1.6.2. I believe everythins is ok before
> the last upgrade. I download it from launchpad.net.

That's quite problematic. I've just reviewed the changelog between 1.5.14 and
1.5.15 and am not seeing anything there which could explain this. Just to be
clear, when the problem happens, does haproxy still serve traffic but at 100%
CPU or does it spin on itself and stop processing traffic ?

I would guess the former, which could be a but in a timeout handling somewhere.

Another point while I'm thinking about it, could it be that sometimes your
processes are reloaded ? I mean, if one of the last request takes 15mn to
complete and you perform a soft reload, the process will not exit until
this request is completed. And by definition, when it completes, it's the
last one in the log before the process tries to leave. In this case you'll
see another process in parallel still doing the job (the new one).

Willy

Re: appsession replacement in 1.6

[Willy Tarreau]

Hi Sylvain,

On Thu, Nov 19, 2015 at 05:04:40PM +0100, Sylvain Faivre wrote:
> >What Aleks meant is that you don't need http-buffer-request as it's
> >only used to process POST data which isn't your case.
> 
> Oh OK. Well, I did not talk about http-buffer-request in the beginning, 
> so I thought Aleks recommended to use it.
> I can see that it is not useful in my case, so I removed it again.

I think he wanted to be on the safest side before knowing well all the
details of your setup. Now that it works, time for cleanup has come :-)

> For future reference, here is my latest config. It has been running for 
> a few hours on our staging servers without a problem :
> 
> backend front
> [stick-table for another purpose]
> [other stuff]
> # sticky session on JSESSIONID
> stick on urlp(jsessionid) table front_jsessionid
> stick on urlp(jsessionid,;) table front_jsessionid
> stick on cookie(JSESSIONID) table front_jsessionid
> stick store-response cookie(JSESSIONID) table front_jsessionid
> 
> backend front_jsessionid
> stick-table type string len 24 size 10m expire 1h peers prod

Thanks that's much appreciated. I wanted to work on a mini-doc to
explain how to replace appsession with stick tables but unfortunately I
failed to find the time to do it before the release. So I appreciate
that you share the result of your findings, it will definitely help
other users.

Thanks to you and Aleks for this :-)
Willy

Re: CPU 100% when waiting for the client timeout

[baiyang]

Hi there,

> when the problem happens, does haproxy still serve traffic but at 100% CPU or 
> does it spin on itself and stop processing traffic ?
No, the proxy service is not hang even in the spike time. you can verify it by 
check "haproxy.log" in my first mail. There are a lot of reqeusts have been 
processed normally even under the busy time.

> could it be that sometimes your processes are reloaded ?
I don't very clear what's your means, If you are suspecting the haproxy 
process: I'm pretty sure there is no any reload when the symptoms appear. 
Again, you can check the pidstat.log and haproxy.log to verify it.

If you mean the backend server: We have been checked the two servers involved 
in this case. And:
1.The server will emit if it has been reloaded or restarted, but there are no 
such records even in a whole day.
2.We have been located the problematic application level session:
[2015-11-19 08:20:52.313756 UTC+0800][Info](audit.fend) user logged on from 
58.246.225.90
  uid= xxx
  number = xxx
  user agent = Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like 
Gecko) Chrome/31.0.1650.63 Safari/537.36
@{tid = xx}
@{nid = 0}
And there is pretty normal just before and after this log record.

3.We have been reviewed our distributed coordinate service (something like 
zookeeper and etcd), and there is no log to show the serivce has any down time 
today.

PS: I remembered the new version of haproxy has some strange statistics maybe 
related to this issue:
http://stackoverflow.com/questions/33168469/whats-the-exact-meaning-of-session-in-haproxy/33540826?noredirect=1#comment55014864_33540826
 

Thanks :-)

--

   Best Regards
  BaiYang
  baiy...@gmail.com
  http://baiy.cn
 < END OF EMAIL >  
 
 
From: Willy Tarreau
Date: 2015-11-20 02:21
To: baiyang
CC: Lukas Tribus; haproxy
Subject: Re: RE: CPU 100% when waiting for the client timeout
Hello,
 
On Fri, Nov 20, 2015 at 12:09:03AM +0800, baiyang wrote:
> > Which release did you use previously that worked fine?
> I'm using 1.5.14 before upgraded to 1.6.2. I believe everythins is ok before
> the last upgrade. I download it from launchpad.net.
 
That's quite problematic. I've just reviewed the changelog between 1.5.14 and
1.5.15 and am not seeing anything there which could explain this. Just to be
clear, when the problem happens, does haproxy still serve traffic but at 100%
CPU or does it spin on itself and stop processing traffic ?
 
I would guess the former, which could be a but in a timeout handling somewhere.
 
Another point while I'm thinking about it, could it be that sometimes your
processes are reloaded ? I mean, if one of the last request takes 15mn to
complete and you perform a soft reload, the process will not exit until
this request is completed. And by definition, when it completes, it's the
last one in the log before the process tries to leave. In this case you'll
see another process in parallel still doing the job (the new one).
 
Willy
 

Connect() failed using unix sockets

[Greg Wilkins]

Hi,

I've been using haproxy 1.6.1 to test the newly added unix domain socket
connector on the Jetty server.

However, under load testing I start to see messages like:

Connect() failed for backend frontend: no free ports.

If I only get a few of these messages, it does not appear to affect the
results. But eventually I get batches of these messages that appear
associated with failed responses.

I have written a test client to connect directly to the jetty unix domain
socket and it works fine with up to 20,000 simultaneous connections and
requests.  I'm only seeing problems via haproxy.

If I use TCP over localhost instead of unix domain socket to talk to the
backend, I get no problems at all.

I'm using ab to generate load and problems start occurring with concurrency
over 350:

ab -n 100 -c 500 -k http://localhost:/


My haproxy configuration is:
global
tune.ssl.default-dh-param 8192
maxconn 1
maxpipes 1
maxsslconn 1

defaults
mode tcp
timeout connect 5000ms
timeout client 5ms
timeout server 5ms

listen frontend
bind :
bind :8843 ssl no-sslv3 crt ./cert.pem ciphers TLSv1.2
#server backend 127.0.0.1:8080
server backend /tmp/jetty.sock send-proxy-v2-ssl-cn


I've tried turning on debug to get more information, but I just get output
like:

01ed:frontend.accept(0005)=013a from [127.0.0.1:53303]
01ee:frontend.accept(0005)=0139 from [127.0.0.1:53302]
01ef:frontend.accept(0005)=0138 from [127.0.0.1:53301]
01f0:frontend.accept(0005)=0137 from [127.0.0.1:53300]
01f1:frontend.accept(0005)=0136 from [127.0.0.1:53299]
01f2:frontend.accept(0005)=0135 from [127.0.0.1:53298]
01f3:frontend.accept(0005)=0134 from [127.0.0.1:53297]
Connect() failed for backend frontend: no free ports.
Connect() failed for backend frontend: no free ports.
Connect() failed for backend frontend: no free ports.
Connect() failed for backend frontend: no free ports.
01b0:frontend.clicls[0177:03aa]
01b0:frontend.closed[0177:03aa]
Connect() failed for backend frontend: no free ports.
Connect() failed for backend frontend: no free ports.
Connect() failed for backend frontend: no free ports.
Connect() failed for backend frontend: no free ports.
01b1:frontend.clicls[0176:0177]
01b1:frontend.closed[0176:0177]
Connect() failed for backend frontend: no free ports.
Connect() failed for backend frontend: no free ports.
Connect() failed for backend frontend: no free ports.
Connect() failed for backend frontend: no free ports.
Connect() failed for backend frontend: no free ports.
Connect() failed for backend frontend: no free ports.

Which is not saying anything to me.

So anybody know what resource "free ports" relates to in the unix domain
socket case?Are there any other debug options to find out more about
what is happening.

I'm running on ubuntu 10.04 with a ulimit of 8 FDs

cheers



-- 
Greg Wilkins  CTO http://webtide.com

Re: Owncloud through Haproxy makes upload not possible

[Igor Cicimov]

On 20/11/2015 7:23 AM, "Piotr Kubaj"  wrote:
>
> On 11/19/2015 17:01, Janusz Dziemidowicz wrote:
> > 2015-11-19 15:45 GMT+01:00 Piotr Kubaj :
> >> Now, about RSA vs ECDSA. I simply don't trust ECDSA. There are quite a
> >> lot of questions about constants used by ECDSA, which seem to be
> >> chosen quite arbitrarily by its creator, which happens to be NSA.
> >> These questions of course remain unanswered. Even respected scientists
> >> like Schneier say that RSA should be used instead (see
> >>
https://www.schneier.com/blog/archives/2013/09/the_nsa_is_brea.html#c167
> >> 5929
> >
> > But ECDSA itself does not contain any constants (see
> > https://en.wikipedia.org/wiki/Elliptic_Curve_Digital_Signature_Algorithm
).
> > Yes, you have to choose domain parameters and most commonly used are
> > NIST ones. But you can also use brainpool curves, which specifically
> > avoid using any arbitrary constants (see
> > http://www.ecc-brainpool.org/download/Domain-parameters.pdf) and they
> > are even defined for TLS (https://tools.ietf.org/html/rfc7027) and
> > apparently supported by latest OpenSSL. Unfortunately not by anything
> > else.
> > OK, anyway that's your preference, I'm not going to argue about ECDSA
or not;)
> >
> >> ). When I'm done setting my HTTP(S) services, I'll simply limit
> >> incoming connections connections on my firewall so DDOS'ing won't be
> >> possible, unless you DDOS my firewall :)
> >
> > I've never said anything about DDoS. In such setup there is no need
> > for distributed DoS. The CPU usage of RSA 8192 is so high that a
> > single shell script running on a single attack machine can kill any
> > server.
> > If you are willing to limit your connection rate on a firewall to a
> > few per second, then fine;)
> >
> > As for your problem. Now that it seems like SSL problem, can you just
> > try with RSA 4096 or 2048? RSA 8192 is really not much tested in most
> > code, so maybe the problem is in fact related.
> >
> Unfortunately, accessing my HTTPS services by only OpenSSL is out of the
> question. Besides, I use LibreSSL and am not sure it supports it, since
> OpenBSD people got rid of quite a lot of unnecessary code.
>
> So I can only choose ECDSA or RSA.
>
> I don't think limiting my connections is a bad idea vs choosing weaker
> RSA. As I said before, I actually expect only a few connections at once.
>
> I've generated RSA 2048 cert with:
> openssl req -x509 -newkey rsa:2048 -keyout haproxy.pem -out haproxy.pem
> -days 3650 -nodes
>
> That is, I didn't use any non-default options, such as SHA512.
> Unfortunately, it doesn't yield any result. I'm now considering
> switching to SSL Pass-through, and configuring HTTPS in each of my WWW
> servers, it may be much quicker considering how long I've been getting
> Haproxy to work.
>
It might be something specific to BSD os causing issues for you since I
haven't heard of anyone complaining about ssl till now. You can also try
latest stable 1.5.15 since I can't see any 1.6 specific feature in your
config.

Re: Echo server in Lua

[Thrawn]

Nope :|. I've increased somaxconn to 512 and tcp_max_syn_backlog to 2048, but 
with one process, 1M requests, concurrency 500, I still get a bunch of 
timeouts, and the longest transaction still takes most of the test (eg 32 
seconds out of a test that takes 35).

Any other obvious possibilities?

On Friday, 20 November 2015, 4:12, Willy Tarreau  wrote:



On Thu, Nov 19, 2015 at 07:00:32AM +, Thrawn wrote:
> Hmm...I haven't tuned anything (this is just my workstation, not a server).
> /proc/sys/net/core/somaxconn 128
> /proc/sys/net/ipv4/tcp_max_backlog doesn't exist
> 
>  /proc/sys/net/ipv4/tcp_max_syn_backlog512

Yep sorry it's this one.

> I guess those are pretty low for trying to thrash an echo server...

Yes they're low even for 16k/s.

> any recommendations?

Start by multiplying both of them by 4. If nothing improves it doesn't
come from there.


Willy

TCP connections rdp

[Douglas Fabiano Specht]

staff,
who would have a sample configuration file to use TCP connections on port
3389 RDP and could share?

-- 

Douglas Fabiano Specht

Cargo 化工产品

[zqso]

  
  

国际DHL（中外运U墩豪）UPS（联合包裹）TNT（天地物B流）FEDAEX(联邦快B递)EMS （中国B邮政）快递服务。创建了一批具有科学管理概念的精良的化工快递服务队G伍，投入到现代化C物B流快递市场。欢迎各位新F老顾客咨询了H解！  
联系人：胡V菲（136Z2169Q4223）   QZQ: 277 453 5791Z
 Raise yourX smiling face, the Sshadow left behind 助力每笔订M单成功起航！！    

  

chemical additives for casting, sealing, winding, bonding of electronic components, automobile parts, transformers,etc.

[Nick]

  

To Whom It May Concern:
 
This is Nicky from Jiaxing Nanyang Wanshixing Chemical Co.,Ltd., our company is located in Jiaxing,zhejiang province,China, we specilize in manufacturing chemical additives-Methyl tetrahydrophthalic anhydride (MTHPA), it's acid anhydride curing agent(hardener), our annual production capacity is 4 tons, is the largest manufacturer  of same type product in production and sales in China.

MTHPA has many excellent properties: high purity, light color, low viscosity, good processing properties, low volatility, low toxicity, low solidifying point, durable at room temperature and so on.

It is widely used in casting, sealing, winding, and bonding of electronic components, automobile parts, transformers, high voltage capacitors, mutual inductors, and the impregnation of electronic components, resistors, hybrid integrated circuits. Moreover, it is used for fiberglass irrigation pipes, oil pipes, water filter boxes, insulative bushing, insulated paint and so on. 

For more information,Please review our website:www.jxnychemical.com
  

If you have any questions, please feel free to contact me at your most convenient time.


Best regards
Nick
Jiaxing Nanyang Wanshixing Chemical Co.,Ltd.
No.458,West Shichang Road,Zhapu Harbor Area,Jiaxing,Zhejiang,China.
MP:+86 181-6732-7733
TEL.:+86  573-8221-6761
E-MAIL:liyf9...@163.com
Website:www.jxnychemical.com


  

   
  
 
  

  


 


  

RE: CPU 100% when waiting for the client timeout

[Lukas Tribus]

Hello!



> Sorry for send it again, I just forgot to provide the attachments. 
> 
> Affect verions: at least 1.5.15 and 1.6.2 
> 
> Here is the related part in my configuration: 
> timeout client 15m # 客户端响应超时 
> timeout client-fin 10s # 对客户端连接完成 TCP 4 次挥手超时 
> timeout connect 5s # HAProxy 向后端 Server 建立连接时的等候超时 
> timeout server 15m # 服务器端响应超时 
> timeout server-fin 5s # 对后端 Server 连接完成 TCP 4 次挥手超时 
> 
> timeout http-keep-alive 5m # http keep-alive 连接空闲（等待下一请求到达）保持超时（对前后端均有效，影响 
> http-reuse 选项） 
> timeout http-request 15s # http 请求 *头部* 接收超时（对前后端均有效） 
> 
> And I found after upgraded to 1.6.2 and 1.5.15, my haproxy server offen 
> using 100% CPU

So you are seeing CPU spikes at 100%, about fifty/fifty between
system (kernel) and usr (haproxy).

Which OS and kernel are you using?
Can you post more configuration (not just timeouts)?
Are you using SSL?
Are you using nbproc?
Can you post the output of haproxy -vv?

You are saying the this happens after an upgrade to 1.6.2 or 1.5.15.
Which release did you use previously that worked fine? Did you
only upgrade haproxy to a new release or did you do any other
changes as well (config changes, OS/kernel changes or tuning)?




Regards,

Lukas

  

Re: appsession replacement in 1.6

[Sylvain Faivre]

Hi Willy,

On 11/18/2015 09:31 PM, Willy Tarreau wrote:

Hi Sylvain,

On Tue, Nov 17, 2015 at 03:09:49PM +0100, Sylvain Faivre wrote:

option http-buffer-request

maybe you should stick on the header ;-)


OK I added "option http-buffer-request", it will help for sure !

What do you mean "stick on the header" ?


Oh well, it seems like I forgot to stick on the request URL parameter,
when passed with a question mark.


What Aleks meant is that you don't need http-buffer-request as it's
only used to process POST data which isn't your case.


Oh OK. Well, I did not talk about http-buffer-request in the beginning, 
so I thought Aleks recommended to use it.

I can see that it is not useful in my case, so I removed it again.




So I just added this to my config :
  stick on urlp(JSESSIONID) table back_jsessionid
  stick store-request urlp(JSESSIONID) table back_jsessionid

It should cover all cases now.


These two are partially redundant, since "stick on" is already a
shortcut for "stick match" + "stick store-request". So you end up
having this :

   stick match urlp(JSESSIONID) table back_jsessionid
   stick store-request urlp(JSESSIONID) table back_jsessionid
   stick store-request urlp(JSESSIONID) table back_jsessionid

Thus you can safely remove the "stick store-request" rule, you won't
notice the difference.



OK, I did that, and also changed urlp(jsessionid) to lower case, since 
our application server passes it in lower case when in URL parameters.


For future reference, here is my latest config. It has been running for 
a few hours on our staging servers without a problem :


backend front
[stick-table for another purpose]
[other stuff]
# sticky session on JSESSIONID
stick on urlp(jsessionid) table front_jsessionid
stick on urlp(jsessionid,;) table front_jsessionid
stick on cookie(JSESSIONID) table front_jsessionid
stick store-response cookie(JSESSIONID) table front_jsessionid

backend front_jsessionid
stick-table type string len 24 size 10m expire 1h peers prod


Best regards,
Sylvain

Re: Owncloud through Haproxy makes upload not possible

[Janusz Dziemidowicz]

2015-11-19 15:45 GMT+01:00 Piotr Kubaj :
> Now, about RSA vs ECDSA. I simply don't trust ECDSA. There are quite a
> lot of questions about constants used by ECDSA, which seem to be
> chosen quite arbitrarily by its creator, which happens to be NSA.
> These questions of course remain unanswered. Even respected scientists
> like Schneier say that RSA should be used instead (see
> https://www.schneier.com/blog/archives/2013/09/the_nsa_is_brea.html#c167
> 5929

But ECDSA itself does not contain any constants (see
https://en.wikipedia.org/wiki/Elliptic_Curve_Digital_Signature_Algorithm).
Yes, you have to choose domain parameters and most commonly used are
NIST ones. But you can also use brainpool curves, which specifically
avoid using any arbitrary constants (see
http://www.ecc-brainpool.org/download/Domain-parameters.pdf) and they
are even defined for TLS (https://tools.ietf.org/html/rfc7027) and
apparently supported by latest OpenSSL. Unfortunately not by anything
else.
OK, anyway that's your preference, I'm not going to argue about ECDSA or not;)

> ). When I'm done setting my HTTP(S) services, I'll simply limit
> incoming connections connections on my firewall so DDOS'ing won't be
> possible, unless you DDOS my firewall :)

I've never said anything about DDoS. In such setup there is no need
for distributed DoS. The CPU usage of RSA 8192 is so high that a
single shell script running on a single attack machine can kill any
server.
If you are willing to limit your connection rate on a firewall to a
few per second, then fine;)

As for your problem. Now that it seems like SSL problem, can you just
try with RSA 4096 or 2048? RSA 8192 is really not much tested in most
code, so maybe the problem is in fact related.

-- 
Janusz Dziemidowicz

Re: RE: CPU 100% when waiting for the client timeout

[baiyang]

These two server are all have the problem:

> Can you post the output of haproxy -vv?
HA-Proxy version 1.5.15 2015/11/01
Copyright 2000-2015 Willy Tarreau 
Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat 
-Wformat-security -Werror=format-security -D_FORTIFY_SOURCE=2
  OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1
Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.3.4
Compression algorithms supported : identity, deflate, gzip
Built with OpenSSL version : OpenSSL 1.0.1 14 Mar 2012
Running on OpenSSL version : OpenSSL 1.0.1 14 Mar 2012
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.12 2011-01-15
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT 
IP_FREEBIND
Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

==
HA-Proxy version 1.6.2 2015/11/03
Copyright 2000-2015 Willy Tarreau 

Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat 
-Wformat-security -Werror=format-security -D_FORTIFY_SOURCE=2
  OPTIONS = USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.3.4
Compression algorithms supported : identity("identity"), deflate("deflate"), 
raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.0.1 14 Mar 2012
Running on OpenSSL version : OpenSSL 1.0.1 14 Mar 2012
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.12 2011-01-15
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with Lua version : Lua 5.3.1
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT 
IP_FREEBIND

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

As you can see, One is 1.5.15 and another is 1.6.2. Because I degraded one of 
them from 1.6.2 to 1.5.15, and the problem still exist.

> Which OS and kernel are you using?
Ubuntu 12.04 LTS Server

> Can you post more configuration (not just timeouts)?
Of course, but other thins in it is quite normal and no computationally 
intensive there, please see the attachment.

> Are you using SSL?
Yes, you can see in the previous mail, one spike is caused by a https request, 
another caused by a plain http request, So I think not ssl's fault here :-)

> Are you using nbproc?
No.

> Which release did you use previously that worked fine?
I'm using 1.5.14 before upgraded to 1.6.2. I believe everythins is ok before 
the last upgrade. I download it from launchpad.net.

> Did you only upgrade haproxy to a new release or did you do any other changes 
> as well (config changes, OS/kernel changes or tuning)?
No, only haproxy is upgraded and this issue exist. Although I've done a 
"apt-get update && apt-get upgrade" while troubleshooting this problem (The 
upgrade is done after the problem already exists).

--

   Best Regards
  BaiYang
  baiy...@gmail.com
  http://baiy.cn
 < END OF EMAIL >  
 
 
发件人： Lukas Tribus
发送时间： 2015-11-19 23:40
收件人： baiyang; haproxy
主题： RE: CPU 100% when waiting for the client timeout
Hello!
 
 
 
> Sorry for send it again, I just forgot to provide the attachments. 
> 
> Affect verions: at least 1.5.15 and 1.6.2 
> 
> Here is the related part in my configuration: 
> timeout client 15m # 客户端响应超时 
> timeout client-fin 10s # 对客户端连接完成 TCP 4 次挥手超时 
> timeout connect 5s # HAProxy 向后端 Server 建立连接时的等候超时 
> timeout server 15m # 服务器端响应超时 
> timeout server-fin 5s # 对后端 Server 连接完成 TCP 4 次挥手超时 
> 
> timeout http-keep-alive 5m # http keep-alive 连接空闲（等待下一请求到达）保持超时（对前后端均有效，影响 
> http-reuse 选项） 
> timeout http-request 15s # http 请求 *头部* 接收超时（对前后端均有效） 
> 
> And I found after upgraded to 1.6.2 and 1.5.15, my haproxy server offen 
> using 100% CPU
 
So you are seeing CPU spikes at 100%, about fifty/fifty between
system (kernel) and usr (haproxy).
 
Which OS and kernel are you using?
Can you post more configuration (not just timeouts)?
Are you using SSL?
Are you using nbproc?
Can you post the output of haproxy -vv?
 
You are saying the this happens after an upgrade to 1.6.2 or 

Re: Owncloud through Haproxy makes upload not possible

[Piotr Kubaj]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256



On 11/19/15 14:41, Janusz Dziemidowicz wrote:
> 2015-11-19 11:13 GMT+01:00 Piotr Kubaj :
>>> 4096 bit DH params will be pretty slow to handshake. Maybe
>>> that's okay in your circumstance though since you seem to be
>>> using this for a personal use and not expecting a high
>>> connection rate. You also have a 8 kbit RSA self signed
>>> certificate and using 256 bit ciphers which increase TLS
>>> overhead.
>> I want it to be secure, and I don't want to touch my settings
>> for quite a while so I just took the strongest algorithms there
>> are, and 2x recommended values for things like private key, or DH
>> params. The hardware is pretty powerful and I've already checked
>> that I don't have a huge load.
> 
> Take note, that increasing RSA size twice reduces number of 
> connections you can accept 10 times or more. For example, my quite
> powerful desktop with recent CPU can accept 973 connections per
> second per core for RSA 2048, 136 connections per second for RSA
> 4096. OpenSSL does not have tests for RSA 8192, but that would be
> in the order of a few connections per second. RSA 8192 is really an
> overkill, it would be possible to DoS your server with a simple
> shell script;) If you want a state of the art cryptography you 
> should probably use ECDSA certificate, it will be both faster and
> more secure.
> 
I've tweaked my haproxy.conf a little, now it looks like this:
global
#ssl-default-bind-options no-sslv3 no-tls-tickets force-tlsv12
#ssl-default-bind-ciphers AES256+EECDH:AES256+EDH
#tune.ssl.default-dh-param 4096
log /var/run/log local0
maxconn 4096
user daemon
group daemon
daemon

defaults
log global
modehttp
option  httplog
option forwardfor
option http-server-close
timeout connect 5000ms
timeout client 5ms
timeout server 5ms


frontend http-in
bind 192.168.11.3:80
acl host_phpsysinfo hdr_dom(host) -i phpsysinfo.anongoth.pl
acl host_owncloud hdr_dom(host) -i owncloud.anongoth.pl
acl host_roundcube hdr_dom(host) -i roundcube.anongoth.pl
acl host_10amd64 hdr_dom(host) -i 10amd64.anongoth.pl
acl host_10armv6 hdr_dom(host) -i 10armv6.anongoth.pl

## figure out which one to use
use_backend phpsysinfo if host_phpsysinfo
use_backend owncloud if host_owncloud
use_backend roundcube if host_roundcube
use_backend 10amd64 if host_10amd64
use_backend 10armv6 if host_10armv6

#   redirect scheme https code 301 if !{ ssl_fc }

frontend https-in
#option httplog
#option forwardfor
#option http-server-close
#option httpclose
#http-response set-header Strict-Transport-Security
"max-age=1600; includeSubDomains; preload;"
#http-response set-header Public-Key-Pins:
"pin-sha256="1Pw5h93NOsPw6j/vaTYl5VvW9cmtuZXtNP3cVz10hKo=";
max-age=15768000; includeSubDomains"
#bind 192.168.11.3:443 ssl crt /usr/local/etc/haproxy.pem ciphers
AES256+EECDH:AES256+EDH force-tlsv12 no-sslv3
bind 192.168.11.3:443 ssl crt /usr/local/etc/haproxy.pem
acl host_phpsysinfo hdr_dom(host) -i phpsysinfo.anongoth.pl
acl host_owncloud hdr_dom(host) -i owncloud.anongoth.pl
acl host_roundcube hdr_dom(host) -i roundcube.anongoth.pl
acl host_10amd64 hdr_dom(host) -i 10amd64.anongoth.pl
acl host_10armv6 hdr_dom(host) -i 10armv6.anongoth.pl

## figure out which one to use
use_backend phpsysinfo if host_phpsysinfo
use_backend owncloud if host_owncloud
use_backend roundcube if host_roundcube
use_backend 10amd64 if host_10amd64
use_backend 10armv6 if host_10armv6

backend 10amd64
server node1 192.168.11.3:81

backend 10armv6
server node1 192.168.11.3:81

backend phpsysinfo
server node1 192.168.11.3:81

backend owncloud
server node1 192.168.11.14:80

backend roundcube
server node1 192.168.1.101:80






With that, I can tell that the issue is about HTTPS section. HTTP
connection through Haproxy goes fine, but HTTPS still has a problem.
I've also got regular Haproxy logs, I'm not sure if they can help
diagnose the problem:
Nov 19 15:04:58 anongoth haproxy[11792]: 46.248.161.165:54997
[19/Nov/2015:15:04:52.559] https-in owncloud/node1 5900/0/0/0/5901 200
319 - -  2/2/0/1/0 0/0 "GET
/core/css/images/ui-bg_flat_35_1d2d44_40x100.png HTTP/1.1"
Nov 19 15:04:58 anongoth haproxy[11792]: 46.248.161.165:54996
[19/Nov/2015:15:04:52.555] https-in owncloud/node1 5911/0/0/1/5912 200
323 - -  2/2/0/1/0 0/0 "GET
/core/css/images/ui-bg_highlight-soft_100_ee_1x100.png HTTP/1.1"
Nov 19 15:05:00 anongoth haproxy[11792]: 46.248.161.165:54997
[19/Nov/2015:15:04:58.459] https-in owncloud/node1 27/0/0/1876/1931
200 1076 - -  3/3/0/1/0 0/0 "POST

Re: Owncloud through Haproxy makes upload not possible

[Janusz Dziemidowicz]

2015-11-19 11:13 GMT+01:00 Piotr Kubaj :
>> 4096 bit DH params will be pretty slow to handshake. Maybe that's
>> okay in your circumstance though since you seem to be using this
>> for a personal use and not expecting a high connection rate. You
>> also have a 8 kbit RSA self signed certificate and using 256 bit
>> ciphers which increase TLS overhead.
> I want it to be secure, and I don't want to touch my settings for
> quite a while so I just took the strongest algorithms there are, and
> 2x recommended values for things like private key, or DH params. The
> hardware is pretty powerful and I've already checked that I don't have
> a huge load.

Take note, that increasing RSA size twice reduces number of
connections you can accept 10 times or more.
For example, my quite powerful desktop with recent CPU can accept 973
connections per second per core for RSA 2048, 136 connections per
second for RSA 4096. OpenSSL does not have tests for RSA 8192, but
that would be in the order of a few connections per second. RSA 8192
is really an overkill, it would be possible to DoS your server with a
simple shell script;) If you want a state of the art cryptography you
should probably use ECDSA certificate, it will be both faster and more
secure.

-- 
Janusz Dziemidowicz