Re: problems with req.ssl_ec_ext
On Mon, May 30, 2016 at 5:24 PM, Nenad Merdanovic wrote:
> Yes, it seems like the case is that the fetch is only called once, when
> there is not enough data in the buffer. You can work around this like:
> tcp-request inspect-delay 5s
> tcp-request content accept if {req.ssl_ec_ext 0 }
Thanks a lot Nenad! That solved my issues! I'll experiment a bit
before trying it out in production environment.
Br
Björn Zettergren
problems with req.ssl_ec_ext
Hi,
I've been playing around with the ECC+RSA certificate on same IP as
described in the haproxy blog at
http://blog.haproxy.com/2015/07/15/serving-ecc-and-rsa-certificates-on-same-ip-with-haproxy/
However, I get unexpected results when testing and I'm thinking that
my problem is with the sample fetching of req.ssl_ec_ext on incoming
requests being inconsistent or haproxy starts processing the request
before enough of the data has been sent. I don't know how to
troubleshoot any further or how to get it working, if it's at all
expected to work "as advertised".
I've reduced the setup to a simple test case, without any certificates
or anything like that. I'm using the following configuration:
(NOTE: Yes, i'm fully aware that this config is not a valid ssl/tls
config! I'm using it to illustrate the problem in frontend tcp-relay
which happens before any certificates get involved)
--- start of haproxy.cfg ---
global
log /dev/log local0
defaults
log global
timeout client 5s
timeout server 5s
timeout connect 5s
frontend tcp-relay
mode tcp
option tcplog
bind *:443
use_backend yes-be-ecc if { req.ssl_ec_ext 1 }
default_backend no-be-ecc
backend yes-be-ecc
mode http
option httplog
server yes-ecc 127.0.0.1:80
backend no-be-ecc
mode http
option httplog
server no-ecc 127.0.0.1:80
--- end of haproxy.cfg ---
Then to test, I'd repeatedly issue the following openssl command and
watch the logs for which backend haproxy selected.
# openssl s_client -tls1_2 -no_ticket -cipher
ECDHE-ECDSA-AES256-GCM-SHA384 -connect 127.0.0.1:443
output from log:
tcp-relay no-be-ecc/ -1/-1/0 187 PR 0/0/0/0/3 0/0
tcp-relay no-be-ecc/ -1/-1/0 187 PR 0/0/0/0/3 0/0
tcp-relay no-be-ecc/ -1/-1/0 187 PR 0/0/0/0/3 0/0
tcp-relay yes-be-ecc/ -1/-1/0 187 PR 0/0/0/0/3 0/0
tcp-relay no-be-ecc/ -1/-1/0 187 PR 0/0/0/0/3 0/0
tcp-relay no-be-ecc/ -1/-1/0 187 PR 0/0/0/0/3 0/0
Mostly it doensn't select the ECC backend, but occationally it does,
about 1 out of 10 times on my VM.
The interesting part comes when i use tcpflow to extract the
ClientHello packet using tcpflow, and then just netcat:ing the same
clienthello message to the server port:
# tcpflow -i lo dst port 443 &
# openssl s_client -tls1_2 -no_ticket -cipher
ECDHE-ECDSA-AES256-GCM-SHA384 -connect 127.0.0.1:443
# fg
run this a bunch of times:
# cat 127.000.000.001.60092-127.000.000.001.00443 | nc 127.0.0.1 443
output from log:
tcp-relay yes-be-ecc/ -1/-1/0 187 PR 0/0/0/0/3 0/0
tcp-relay yes-be-ecc/ -1/-1/0 187 PR 0/0/0/0/3 0/0
tcp-relay yes-be-ecc/ -1/-1/0 187 PR 0/0/0/0/3 0/0
tcp-relay no-be-ecc/ -1/-1/0 187 PR 0/0/0/0/3 0/0
tcp-relay yes-be-ecc/ -1/-1/0 187 PR 0/0/0/0/3 0/0
tcp-relay yes-be-ecc/ -1/-1/0 187 PR 0/0/0/0/3 0/0
tcp-relay yes-be-ecc/ -1/-1/0 187 PR 0/0/0/0/3 0/0
tcp-relay yes-be-ecc/ -1/-1/0 187 PR 0/0/0/0/3 0/0
tcp-relay yes-be-ecc/ -1/-1/0 187 PR 0/0/0/0/3 0/0
tcp-relay yes-be-ecc/ -1/-1/0 187 PR 0/0/0/0/3 0/0
tcp-relay yes-be-ecc/ -1/-1/0 187 PR 0/0/0/0/3 0/0
tcp-relay yes-be-ecc/ -1/-1/0 187 PR 0/0/0/0/3 0/0
tcp-relay yes-be-ecc/ -1/-1/0 187 PR 0/0/0/0/3 0/0
Then it seem to work most of the time, only 1 of 20 are sent to the
wrong backend.
Just for fun i tried pushing the file at different bitrates to
haproxy, and at ~1500bps, I would start seeing failures.
mostly works;
# pv -q -L 1700 127.000.000.001.60092-127.000.000.001.00443 | nc 127.0.0.1 443
always fails:
# pv -q -L 1400 127.000.000.001.60092-127.000.000.001.00443 | nc 127.0.0.1 443
I'm running on Ubuntu Trusty, latest stable haproxy 1.6.5 from Vincent
Bernats PPA.
Am I right to assume that fetches such as req.ssl_ec_ext which is used
to build conditions should always work?
Have I made catastrophic config-errors in my test-case that I'm
totally missing?
Please advice :-)
Best regards
Björn Zettergren
Re: HAproxy and ftp_put response "504 Timeout"
On Mon, May 16, 2016 at 4:13 PM, Info (ITpartner.ee) wrote: > So FirewallD ports are opened. Doing the same ftp_put via php in active or > passive mode returns 504 Gateway timeout. > That’s why I thought mby haproxy doesn't let FTP server response back in to > server. > > Mby you have any other suggestions in this case? Not knowing how your script works or any more details, i'm guessing that you're calling it through haproxy via port 80 on a browser, and then it tries to upload something remotely and won't give any feedback to your client until upload is finished. If this is the case, you might need to tune "timeout server". Set at value that's higher than the time it takes for the script to upload your files to the remote server. But now i'm only guessing. If i'm wrong, please share more details about the script and how you call it etc. /Björn
Re: HAproxy and ftp_put response "504 Timeout"
Hi Juri, I suspect you have a problem with the FTP protocol and the clients active / passive mode. You need to forward the appropriate ports in your firewall/nat at the node running haproxy. Quick google suggested this read; http://slacksite.com/other/ftp.html This is not a haproxy problem. Best regards Björn On Mon, May 16, 2016 at 3:35 PM, Info (ITpartner.ee) wrote: > Well yes, internal php script makes a call to some outside FTP server, > looking like: > > > > |--| > > | app_back (192.168.11.254) - > HAproxy (with public IP)| - > Some FTP > hosting > > |--| > > > > Could you please clarify what do you mean under “loadbalance port 21”? > > > > From: Andrew Smalley [mailto:asmal...@loadbalancer.org] > Sent: Monday, May 16, 2016 4:27 PM > To: Info (ITpartner.ee) > Cc: HAProxy > Subject: Re: HAproxy and ftp_put response "504 Timeout" > > > > Juri > > Your welcome. > > I made an assumption that you would need to also loadbalance port 21 for > ftp. > > Is your php script making a call to an external service or is it behind > haproxy? > > > Regards > > Andrew Smalley > > Loadbalancer.org > > http://www.loadbalancer.org > > > > On 16 May 2016 at 14:24, Info (ITpartner.ee) wrote: > > Hello Andrew, > > > > Thanks for fast reply! > > I have already tried similar configuration and tried yours, but I get > “ERR_CONNECTION_REFUSED” on any http request to this server. > > > > After removing ftp_front and ftp_back + haproxy restart, all fine again. > > > > // Juri > > > > From: Andrew Smalley [mailto:asmal...@loadbalancer.org] > Sent: Monday, May 16, 2016 4:10 PM > To: Info (ITpartner.ee) > Cc: HAProxy > Subject: Re: HAproxy and ftp_put response "504 Timeout" > > > > Hello Juri > > You say your PHP Script uses FTP Protocol, so you will need to add another > VIP for ftp on port 21 in TCP Mode as shown below > > frontend app_front > >bind *:80 > >default_backend admin_back > > > > backend admin_back > >mode http > >balance roundrobin > >server admin 192.168.11.254:80 check > > frontend ftp_front > > > >bind *:21 > >default_backend ftp_back > > > > backend ftp_back > >mode tcp > >balance roundrobin > >server admin 192.168.11.254:21 check > > > Regards > > Andrew Smalley > > Loadbalancer.org > > http://www.loadbalancer.org > > > > On 16 May 2016 at 14:02, Info (ITpartner.ee) wrote: > > Hello, > > > > Really need a help to solve this issue. We have PHP ftp file upload script. > After project migration from regular hosting under HAproxy as a load > balancer, our php file upload script stopped working. > > We receive “504 Gateway Time-out. The server didn't respond in time.” error > after ~30 seconds. Increasing keep-alive time or php execution time makes no > effect. > > By checking php code line by line we realized that ftp_connect, ftp_login – > works fine, but it gets an error after ftp_put request. > > Our haproxy conf looks like this: > > > > frontend app_front > >bind *:80 > >default_backend admin_back > > > > backend admin_back > >mode http > >balance roundrobin > >server admin 192.168.11.254:80 check > > > > Tried “ftp –n” connection directly from server – all fine. Turning off > FirewallD service on both haproxy and “admin_back” server makes no > difference. > > Can you give us any suggestions? > > > > BR, > > Juri > > > >
Re: Haproxy failing to start after VM move (Debian 8 [systemd])
Hi Have you verified that your new VM host exposes the same cpu features? If your haproxy binary is compiled expecting one set of features, it might not work well if that suddenly disappear. (for example if you moved from AMD to INTEL cpu, or your hypervisor exposes different CPU model) /Björn Zettergren On Mon, Mar 7, 2016 at 1:08 PM, SL wrote: > Hi, > > I hope someone can offer some insight here, as I'm currently at a bit of a > loss, and our service is down. We are running HAproxy 1.6.1 on Debian 8 > virtuial machines. I was moving our 2 instances to a different size VM, > however, when I rebooted they did not come back up. > > I can't even run the haproxy binary file. Here's what I get when I try: > > ./haproxy > Illegal Instruction > > sudo ./haproxy > [no output] > > ./haproxy -v > Illegal Instruction > > sudo/haproxy -v > [no output] > > Same thing if I try to check the config with -c -f (though I don't think > this is a config issue). > > Here's what I have in kern.log: > > Mar 7 11:41:41 rproxyws1 kernel: traps: haproxy[4031] trap invalid opcode > ip:4843a0 sp:7fff51856ac8 error:0 in haproxy[40+cf000] > > One thing that does seem to have happened is a kernel update during the move > (from 4.1.5 to 4.4.0), but I've manually renerted to the old kernel, but the > new problems remain. > > Does anyone have any ideas!? > > Thank you! > > Scott >
Re: http-response redirect
> http-response redirect code 302 location https://blabla if { status 404 }
> statement in my haproxy 1.5.15 config but on reload I get:
If i make proper sense of the documentation; the "http-response" does
not support "redirect" in 1.5.
https://cbonte.github.io/haproxy-dconv/configuration-1.5.html#4-http-response
But as you noted, the source-code contain the "redirect" in list of
expected options. I'm no coder, so i'm not sure i read the source
right, but it seems like the functions for "redirect" in
"http-response" are not available in 1.5.15 but the errormessage
contain it as a possibility, and it seem to be available in 1.6.
I suspect that this is a bug in 1.5 (either error message is wrong, or
the redirect-functions are not properly implemented).
/Björn
Re: SSL acceleration
Hi Eric, If you use a hardware device supported by openssl library you'll have hardware acceleration, for example AES-NI extension is available on recent cpu's and recent versions of openssl. I don't know about your Coleto creek device, but i'm sure you can check with openssl :) /Björn On Fri, Jan 29, 2016 at 5:56 PM, Eric Chan wrote: > Hi HAproxy team, > > > > Is there a plan to add HW acceleration to your SSL proxy? > > I am thinking of using HAproxy with Intel Coleto Creek in asynchronous mode, > wonder if anyone has done the patch work that needs to make that work. > > > > Thanks, > > Eric > > This email and any attachments thereto may contain private, confidential, > and/or privileged material for the sole use of the intended recipient. Any > review, copying, or distribution of this email (or any attachments thereto) > by others is strictly prohibited. If you are not the intended recipient, > please contact the sender immediately and permanently delete the original > and any copies of this email and any attachments thereto.
Re: Haproxy Config clean up
Hi Mel, change sensitive information like passwords (and possibly ip-addresses) to generic ones, and put your config on pastebin and send us the link, and i'm sure someone can give you feedback on it from here. /Björn On Thu, Jan 28, 2016 at 1:15 AM, Mela Luca wrote: > I am looking for somebody to help me clean up my haproxy config and > recommend some improvements, any takers. Willing to pay =). > > Mel
Re: Upgrade from 1.4 -> 1.6, any gotchas?
On Wed, Oct 21, 2015 at 7:14 PM, SL wrote: > I'll be doing an upgrade from 1.4 to 1.6 tomorrow. Just wondering if there > are any changed defaults, breaking changes, anything like that? Or should > my config work as before? Haproxy 1.5 changed the default connection mode if you use http. Up to version 1.5-dev21 "option tunnel" was the default, now it's "option http-keep-alive". https://cbonte.github.io/haproxy-dconv/configuration-1.5.html#4 This bit me a bit, when I upgraded from 1.4 to 1.5, as the tunnel mode works a bit different. But with settings "option http-keep-alive" and "option prefer-last-server" specified, everthing was fine. I'm not sure if there are other changed defaults that you'd need to look out for, I haven't read through the manual for 1.6 yet. That's a good startingpoint though, verify your config against it, and you should probably be fine :-) Good luck. /Björn
Re: Statistics in multi-process mode
On Tue, Apr 14, 2015 at 9:35 AM, hiepnv wrote: > Therefore, I wrote code to support statistics in multi-process mode and want > to add this to haproxy in future releases. > And how can I contribute my code? I don't know if there are other works in progress to support what you're looking to share, but there's general notes available in section 5 of the README in the haproxy sources "How To Contribute": https://github.com/haproxy/haproxy The first step would probably be to submit a patch for general review/comments :-) Best Regards Björn Zettergren
Re: CPU usage increase / bad performance after upgrading haproxy from 1.4.25 to 1.5.11
> I've just re-upgraded my haproxy instances to 1.5.11 and added the > "option http-tunnel" as you suggested. I'm waiting for things to > stabilize before jumping to conclusions, but it looks a bit better at > the moment. > I'll return with verdict about performance in a while after monitoring > further! So far it looks a lot better with the "option http-tunnel" option set after upgrading to 1.5.11 at ~12:00 today. I also manage to dig out a combined graph in custom date format to get a better view of tcp_connections, something that i completely overlooked yesterday, and the default "option http-keep-alive" seems to have caused some havoc to my amount of established connections which explain the increased cpu-usage: http://imgur.com/EToNrhS I also noted that we seem to have rather unevenly distributed clients with our rrdns as the 03 server recieves more reqeusts overall, but that's another story and have noting to do with 1.4 to 1.5 upgrade. Thanks for your help so far Lukas, I really appreciate it. I should also have listen better to PiBa-NL in #haproxy@FreeNode who suggested the same change to look at yesterday evening. :) /Björn
Re: CPU usage increase / bad performance after upgrading haproxy from 1.4.25 to 1.5.11
>> Setting: >> option http-tunnel > Can you provide the output of "haproxy -vv" please? Hi Lukas, I've just re-upgraded my haproxy instances to 1.5.11 and added the "option http-tunnel" as you suggested. I'm waiting for things to stabilize before jumping to conclusions, but it looks a bit better at the moment. Here's the output of "haproxy -vv" on 1.4.25: http://paste.ubuntu.com/10550216/ and here's the output of the same for 1.5.11: http://paste.ubuntu.com/10550222/ I note there's different polling, sepoll vs epoll, but I don't know what that means (yet) :) I'll return with verdict about performance in a while after monitoring further! /Björn
CPU usage increase / bad performance after upgrading haproxy from 1.4.25 to 1.5.11
Hi haproxy-list! I've got 3 active haproxy servers (multiple ip dns rr + keepalived) fronting 15 backend-servers. The haproxies have a combined session rate at ~10-14k, at ~500mbps. We have a low latency application which does programmatic bidding/buying, where every ms counts, thus CPU-utilization is very important factor in keeping it down. We are setting a CPU affinity mask to the process to keep it pinned to a single core, and in haproxy 1.4.25 it uses about 40-50% CPU. Yesterday i upgraded haproxy from 1.4.25 to 1.5.11 and noticed increased CPU-usage, which caused problems for us. Here is CPU metrics graph from yesterday, upgrade to 1.5 took place before 12, and I downgraded to 1.4 in the evening: http://imgur.com/jrVvHlw . This increase represent going from 40-50% CPU to 94-100%. Here's also some contextual weekly graph of the same metric: http://imgur.com/o4bgtM5 Finally, here's a version of our config (i've changed names/ips/passwords, but nothing else): http://paste.ubuntu.com/10548978/ I did see something in the graphs regarding fd_inuse: http://imgur.com/hTEOfMq but I'm uncertain as what to make of it. As said, the same config runs smoothly with haproxy 1.4. I have not been able to reproduce the behavior in my lab-environment, and I would very much appreciate pointers as how to troubleshoot this further, and where to look. I have a feeling that haproxy 1.5 have default values which I've not taken into consideration that might cause this. Please advice! :-) Best regards Björn Zettergren
