Re: haproxy resolvers, DNS query not send / result NXDomain not expected

[Baptiste]

On Tue, Sep 8, 2015 at 7:58 AM, Baptiste  wrote:
>>> Hi,
>>>
>>> I wonder why the code send the TCP port in the DNS query...
>>> I'm currently installing an opnsense and I'll try to reproduce the
>>> problem.
>>>
>>> I've not used FreeBSD since 5.4 version :)
>>>
>>> Baptiste
>>
>> Hi Baptiste,
>>
>> it seems ipv4 and ipv6 are handled differently regarding the removal of the
>> port in server.c and this is apparently done after the initial dns check has
>> already been performed.?. as it knows to treat 'google.com' differently from
>> 'nu.nl'
>>
>> /* save hostname and create associated name resolution */
>> switch (sk->ss_family) {
>> case AF_INET: {
>> /* remove the port if any */
>> Alert("\n AF_INET remove the port if any: %s", args[2]);
>> char *c;
>> if ((c = rindex(args[2], ':')) != NULL) {
>> newsrv->hostname = my_strndup(args[2], c - args[2]);
>> }
>> else {
>> newsrv->hostname = strdup(args[2]);
>> }
>> }
>> break;
>> case AF_INET6:
>> Alert("\n AF_INET6   args 2  : %s", args[2]);
>> newsrv->hostname = strdup(args[2]);
>> break;
>> default:
>> goto skip_name_resolution;
>> }
>> Alert("\nparse_server newsrv->hostname: %s", newsrv->hostname);
>>
>> Result:
>>  AF_INET6   args 2  : www.google.com:80[ALERT] 249/231349
>> (38431) :
>> parse_server newsrv->hostname: www.google.com:80[ALERT] 249/231349 (38431) :
>>  AF_INET remove the port if any: nu.nl:80[ALERT] 249/231349 (38431) :
>> parse_server newsrv->hostname: nu.nl[ALERT] 249/231349 (38431) :
>>
>> Though the tricky part i guess is that ipv6 can contain ':' in a address to
>> but not in a hostname i would think.. Maybe the question should be why have
>> the different treatment at all?
>>
>> I would expect this to not occur on FreeBSD systems.?
>>
>> Hope it helps investigate the issue.
>>
>> PiBa-NL
>
>
> Hi Piba,
>
> You're right and I think this bug is not linked to FreeBSD at all :)
> I think checking port1 and port2 variables at this moment of the code
> is the best way to go instead of relying on ss_family.
> I'll send you a patch later today for testing purpose.
>
> Baptiste


Hi Piba,

I was able to reproduce the bug on my linux machine (that said, I now
have a nice Freebsd 10.2 running in a virtualbox).
Please apply the patch in attachment and tell me whether it fixes the
issue on your side.

Here is new behavior:
09:28:18.097616 IP 10.0.3.20.40165 > 10.0.1.2.53: 20646+ ?
www.google.com. (32)
09:28:18.098140 IP 10.0.1.2.53 > 10.0.3.20.40165: 20646 1/0/0 
2a00:1450:4007:805::1014 (60)
09:28:18.098852 IP 10.0.3.20.38413 > 8.8.8.8.53: 61174+ ANY?
www.google.com. (32)
09:28:18.108185 IP 8.8.8.8.53 > 10.0.3.20.38413: 61174 15/0/0 A
159.180.253.42, A 159.180.253.49, A 159.180.253.30, A 159.180.253.53,
A 159.180.253.38, A 159.180.253.57, A 159.180.253.59, A
159.180.253.34, A 159.180.253.27, A 159.180.253.45, A 159.180.253.23,
A 159.180.253.15, A 159.180.253.19, A 159.180.253.29, A 159.180.253.44
(272)

Baptiste
From c5eb4a02ecd6392d727ad7a08dbe483358e52643 Mon Sep 17 00:00:00 2001
From: Baptiste Assmann 
Date: Tue, 8 Sep 2015 09:28:39 +0200
Subject: [PATCH 11/11] MINOR: FIX: hostname parsing error lead TCP port to be
 part of hostname sent in DNS resolution

A corner case could lead configuration parsing of server's hostname to
use the TCP port as part of the hostname. This happens when the first
resolution from libc points to an IPv6 address.

The patch in attachement fix this by relying of the existence of TCP
port instead of IP address family before saving server's hostname.
---
 src/server.c | 23 ---
 1 file changed, 8 insertions(+), 15 deletions(-)

diff --git a/src/server.c b/src/server.c
index fed180b..d364b5b 100644
--- a/src/server.c
+++ b/src/server.c
@@ -936,23 +936,16 @@ int parse_server(const char *file, int linenum, char **args, struct proxy *curpr
 			}
 
 			/* save hostname and create associated name resolution */
-			switch (sk->ss_family) {
-			case AF_INET: {
-/* remove the port if any */
+			if (!port1 && !port2) {
+/* no ports set, means no ':' are used to split IP from PORT */
+newsrv->hostname = strdup(args[2]);
+			}
+			else {
+/* port set, means ':' is used to split IP from PORT
+ * We want to copy hostname part only */
 char *c;
-if ((c = rindex(args[2], ':')) != NULL) {
+if ((c = rindex(args[2], ':')) != NULL)
 	newsrv->hostname = my_strndup(args[2], c - args[2]);
-}
-else {
-	newsrv->hostname = strdup(args[2]);
-}
-			}
-break;
-			case AF_INET6:
-newsrv->hostname = strdup(args[2]);
-break;
-			default:
-goto skip_name_resolution;
 			}
 
 			/* no name resolution if an IP has been 

7.8USD----720P AHD Camera

[Grace]

 
 

Grace Le

Sales
manager

 

Skype:
graceliyan09 

whatsapp: +86-136 3284
7386Mobile:+86 136 3284 7386

Email:
gr...@aopvision.com 

**SHENZHEN
ANNIDIGITAL TECHNOLOGY CO.,LTD. (aopvision)Add:2F, D Block, No.7
Building East, Shangxue Hi-tech Industrial Park,Buji Town, Longgang
District,Shenzhen, China　　PC:518129Tel: 
+86-755-89392688-8218 Fax: +86 755-89392553Website: www.aopvision.com  Technical Support:Skype:
aopvision_support02
 

HAProxy flow control

[ruoyu Zhang]

Hi,

I've read the HAProxy 1.5 Configuration Manual, and I find restrictions for
connection numbers, session numbers, etc. However, I notice that there is
not any restriction to control the flow rate as they focus more on the
numbers rather than the flow rate. I know that iptables can realize this
function but I this it will be more convenient if HAProxy can provide such
kind of service. Will this be in the blueprint of HAProxy future version?

Best Regards,

Ruoyu Zhang

SSL errors with HAProxy

[Brendan Kearney]

i am not sure what i am doing wrong, but i keep getting errors in my 
browser when trying to browse to my site.  i just moved from an old OS 
and HAProxy instance to current, and may have issues with config 
directives to work out.  please be patient :)


just about every third request works.  otherwise i get "Error code: 
ssl_error_rx_record_too_long" errors in firefox.  if i try to reload the 
page, it takes a couple tries, but does finally load.  i have no idea 
how the error exists for some of the requests but not for all. relevant 
info below.  any help is appreciated.  any other info needed available 
upon request:


version: 1.5.14

haproxy.cfg (edited for relevance, brevity):

global
#debug
daemon
log localhost local1 notice
log-send-hostname router
#uid 996
#gid 995
maxconn 1024
pidfile /var/run/haproxy.pid
stats socket /var/run/haproxy.sock level admin
stats maxconn 2
tune.ssl.default-dh-param 2048

defaults
balance leastconn

log global

mode http

option httplog
option http-server-close
option forwardfor except 127.0.0.0/8

stats enable
stats hide-version
stats refresh 5s
stats scope   .
stats show-legends
stats uri /admin?stats

timeout http-request10s
timeout queue   1m
timeout connect 10s
timeout client  1m
timeout server  1m
timeout http-keep-alive 10s
timeout check   10s

listen https 192.168.120.2:443
bind 192.168.120.2:443 ssl crt /etc/haproxy/www.pem
server www1 192.168.88.1:80
server www2 192.168.88.2:80

SSL and Piranha conversion

[Daniel Zenczak]

Hello All,
First time caller, short time listener. So this is the deal.  
My organization was running a CentOS box with Piranha on it to work as our load 
balancer between our two web servers.  Well the CentOS box was a Gateway 
workstation from 2000 and it finally gave up the ghost.  I have never been in 
that box and I have been scrambling to put something together.  We are now just 
using one of the servers and everything is pointed at it from the firewall.  So 
here is my problems, concerns and other stuff.

1)  I spun up a Ubuntu server in one of our virtual servers.  I am familiar 
with the OS (kinda), so that is what I went with.  I want to us this as the new 
Load Balancer, so I went with Haproxy.

2)  I know nothing about configuring a load balancer.

3)  How do I switch the configs from Piranha to Haproxy?

4)  What do I do with our Https stuff and SSL certs?

5)  How do I test my configs with little interruption to my production 
sites? It is kinda important.

Thanks for any help you guys can give me.

Lost

Re: SSL and Piranha conversion

[Jonathan Matthews]

On 8 Sep 2015 20:07, "Daniel Zenczak"  wrote:
>
> Hello All,
>
> First time caller, short time listener. So this is the
deal.  My organization was running a CentOS box with Piranha on it to work
as our load balancer between our two web servers.  Well the CentOS box was
a Gateway workstation from 2000 and it finally gave up the ghost.

May I suggest you reconsider migrating your hardware and software at the
same time, both whilst under pressure? It will be massively simpler to
install your preexisting choice of (known "good") software on your new
hardware.

Jonathan

Re: SSL and Piranha conversion

[Malcolm Turnbull]

Piranha is a front end for LVS (layer 4 load balancing)
So I'm assuming that all your Piranha box was doing was forwarding
port 443 & 80 to your two servers...

So just set up HAProxy in TCP mode for port 80 & 443.

Test it , and then when you are happy point your DNS at it.



On 8 September 2015 at 20:23, Jonathan Matthews  wrote:
> On 8 Sep 2015 20:07, "Daniel Zenczak"  wrote:
>>
>> Hello All,
>>
>> First time caller, short time listener. So this is the
>> deal.  My organization was running a CentOS box with Piranha on it to work
>> as our load balancer between our two web servers.  Well the CentOS box was a
>> Gateway workstation from 2000 and it finally gave up the ghost.
>
> May I suggest you reconsider migrating your hardware and software at the
> same time, both whilst under pressure? It will be massively simpler to
> install your preexisting choice of (known "good") software on your new
> hardware.
>
> Jonathan



-- 
Regards,

Malcolm Turnbull.

Loadbalancer.org Ltd.
Phone: +44 (0)330 1604540
http://www.loadbalancer.org/

RE: SSL and Piranha conversion

[Daniel Zenczak]

Malcolm,
The Piranha gui had some configurations about Virtual IPs and I am not 
sure how that works or how it is different than HAProxy.  The firewall had some 
rules that pointed website requests to the virtual ips.  

Daniel
-Original Message-
From: Malcolm Turnbull [mailto:malc...@loadbalancer.org] 
Sent: Tuesday, September 8, 2015 2:55 PM
To: Jonathan Matthews 
Cc: Daniel Zenczak ; haproxy 
Subject: Re: SSL and Piranha conversion

Piranha is a front end for LVS (layer 4 load balancing)
So I'm assuming that all your Piranha box was doing was forwarding port 443 & 
80 to your two servers...

So just set up HAProxy in TCP mode for port 80 & 443.

Test it , and then when you are happy point your DNS at it.



On 8 September 2015 at 20:23, Jonathan Matthews  wrote:
> On 8 Sep 2015 20:07, "Daniel Zenczak"  wrote:
>>
>> Hello All,
>>
>> First time caller, short time listener. So this is 
>> the deal.  My organization was running a CentOS box with Piranha on 
>> it to work as our load balancer between our two web servers.  Well 
>> the CentOS box was a Gateway workstation from 2000 and it finally gave up 
>> the ghost.
>
> May I suggest you reconsider migrating your hardware and software at 
> the same time, both whilst under pressure? It will be massively 
> simpler to install your preexisting choice of (known "good") software 
> on your new hardware.
>
> Jonathan



--
Regards,

Malcolm Turnbull.

Loadbalancer.org Ltd.
Phone: +44 (0)330 1604540
http://www.loadbalancer.org/

Re: haproxy resolvers, DNS query not send / result NXDomain not expected

[PiBa-NL]

Op 8-9-2015 om 17:39 schreef Baptiste:

Hi Piba,

Finally, Willy fixed it in a different (and smarter) way:
http://git.haproxy.org/?p=haproxy.git;a=commit;h=07101d5a162a125232d992648a8598bfdeee3f3f

Baptiste

Hi Baptiste,

Just compiled latest snapshot + the list of patches from today and now 
it works good for me.


For some reason 1 time it failed to start while testing, i suspect due 
to dns not replying properly at that moment. That produced the following 
message.
[ALERT] 250/181319 (92688) : parsing [/var/haproxy.cfg:27] : 'server 
testsite2' : invalid address: 'nu.nl' in 'nu.nl:80'
Tried again a few seconds later, and it started without issue. I suppose 
the init-addr could have solved that but i read its on the todo list (at 
least when dev3 was announced). Anyway i could not reproduce the issue 
after starting a tcpdump to see what was going on. Probably nothing.


p.s.
Dont forget to add the patch "get_addr_len(>addr)" Remi 
created ;) its not yet in todays list of dns patches. But maybe im just 
a bit to eager now :) .


Keep up the good work!
Thanks.
PiBa-NL

Re: SSL and Piranha conversion

[Willy Tarreau]

On Tue, Sep 08, 2015 at 09:53:24PM +0100, Malcolm Turnbull wrote:
> Daniel,
> 
> All load balancers work in roughly the same way:
> 
> You have a Virtual IP on the load balancer that the clients talk to,
> and the load balancer is configured to talk to multiple Real
> IPs/Backend Servers.
> Your old config probably had one VIP for HTTP and one for HTTPS.
> 
> HAProxy is very easy but you will need to read the manual or one of
> the many blogs talking about how to use it.
> Once you have studied / installed / configured / tested / understood -
> then if necessary come back to the list for help.

Yep that's the idea. I could suggest that you take a look at the architecture
manual in doc/. It's quite old (2006) but will give you some quick hints
as to the config you need for various use cases. You may recognize some
elements that your old setup was doing and this will help you start with
a simple but working config.

Willy

Re: SSL and Piranha conversion

[Jonathan Matthews]

On 8 September 2015 at 20:56, Daniel Zenczak  wrote:
> Hello Jonathan,
>
> Thank you for the response.  That old gateway workstation is
> not going to be used anymore (the HDDs failed on it and the RAID board
> didn’t warn/detect/tell us when it happened).  I have spun up Ubuntu Server
> inside one of our Virtual Servers to act as the new Load Balancer.  Is this
> what you mean by migrating the hardware as well as the software?

[on-list reply]

Daniel -

You have to swap out your hardware because it failed.
You don't have to swap out your software as it has not failed.

Whilst a move to HAProxy is a great plan, I would not be doing it
whilst trying to fix your web servers' redundancy and bringing both
web servers back into service.

My professional advice in your situation would be to change the
minimum number of things necessary to restore resilient service, which
in this case sounds like only your hardware - whether you fix it by
replacing the hardware or by virtualising the server.

I would not include swapping Piranha for HAProxy and CentOS for Ubuntu
in this work. I'd do both of those later.

HTH,
Jonathan

Re: SSL and Piranha conversion

[Malcolm Turnbull]

Daniel,

All load balancers work in roughly the same way:

You have a Virtual IP on the load balancer that the clients talk to,
and the load balancer is configured to talk to multiple Real
IPs/Backend Servers.
Your old config probably had one VIP for HTTP and one for HTTPS.

HAProxy is very easy but you will need to read the manual or one of
the many blogs talking about how to use it.
Once you have studied / installed / configured / tested / understood -
then if necessary come back to the list for help.



On 8 September 2015 at 20:59, Daniel Zenczak  wrote:
> Malcolm,
> The Piranha gui had some configurations about Virtual IPs and I am 
> not sure how that works or how it is different than HAProxy.  The firewall 
> had some rules that pointed website requests to the virtual ips.
>
> Daniel
> -Original Message-
> From: Malcolm Turnbull [mailto:malc...@loadbalancer.org]
> Sent: Tuesday, September 8, 2015 2:55 PM
> To: Jonathan Matthews 
> Cc: Daniel Zenczak ; haproxy 
> Subject: Re: SSL and Piranha conversion
>
> Piranha is a front end for LVS (layer 4 load balancing)
> So I'm assuming that all your Piranha box was doing was forwarding port 443 & 
> 80 to your two servers...
>
> So just set up HAProxy in TCP mode for port 80 & 443.
>
> Test it , and then when you are happy point your DNS at it.
>
>
>
> On 8 September 2015 at 20:23, Jonathan Matthews  
> wrote:
>> On 8 Sep 2015 20:07, "Daniel Zenczak"  wrote:
>>>
>>> Hello All,
>>>
>>> First time caller, short time listener. So this is
>>> the deal.  My organization was running a CentOS box with Piranha on
>>> it to work as our load balancer between our two web servers.  Well
>>> the CentOS box was a Gateway workstation from 2000 and it finally gave up 
>>> the ghost.
>>
>> May I suggest you reconsider migrating your hardware and software at
>> the same time, both whilst under pressure? It will be massively
>> simpler to install your preexisting choice of (known "good") software
>> on your new hardware.
>>
>> Jonathan
>
>
>
> --
> Regards,
>
> Malcolm Turnbull.
>
> Loadbalancer.org Ltd.
> Phone: +44 (0)330 1604540
> http://www.loadbalancer.org/



-- 
Regards,

Malcolm Turnbull.

Loadbalancer.org Ltd.
Phone: +44 (0)330 1604540
http://www.loadbalancer.org/

Re: TCP_NODELAY in tcp mode

[Dmitry Sivachenko]

> On 30 авг. 2015 г., at 22:29, Willy Tarreau  wrote:
> 
> On Fri, Aug 28, 2015 at 11:40:18AM +0200, Lukas Tribus wrote:
 Ok, you may be hitting a bug. Can you provide haproxy -vv output?
 
>>> 
>>> 
>>> What do you mean? I get the following warning when trying to use this
>>> option in tcp backend/frontend:
>> 
>> Yes I know (I didn't realize you are using tcp mode). I don't mean the
>> warning is the bug, I mean the tcp mode is supposed to not cause any
>> delays by default, if I'm not mistaken.
> 
> You're not mistaken, tcp_nodelay is unconditional in TCP mode and MSG_MORE
> is not used there since we never know if more data follows. In fact there's
> only one case where it can happen, it's when data wrap at the end of the
> buffer and we want to send them together.
> 


Hello,

yes, you are right, the problem is not TCP_NODELAY.  I performed some testing:

Under low network load, passing TCP connection through haproxy involves almost 
zero overhead.
When load grows, at some point haproxy starts to slow things down.

In our testing scenario the application establishes long-lived TCP connection 
to server and sends many small requests.
Typical traffic at which adding haproxy in the middle causes measurable 
slowdown is ~30MB/sec, ~100kpps.

haproxy process CPU usage is about 15-20%.

Re: halog Makefile

[Dmitry Sivachenko]

> On 8 сент. 2015 г., at 17:26, Dmitry Sivachenko  wrote:
> 
> Hello,
> 
> Can you please apply the following patch to contrib/halog/Makefile not to 
> override $CC (on some systems compiler is called "cc", not "gcc"):
> 
> --- Makefile.orig   2015-09-08 17:15:37.423168000 +0300
> +++ Makefile2015-09-08 17:14:57.938196000 +0300
> @@ -1,7 +1,7 @@
> EBTREE_DIR = ../../ebtree
> INCLUDE  = -I../../include -I$(EBTREE_DIR)
> 
> -CC   = gcc
> +CC   ?= gcc
> 
> # note: it is recommended to also add -fomit-frame-pointer on i386
> OPTIMIZE = -O3
> 


Oh, please forget this: the same CC is used in all makefiles, it is simpler to 
redefine CC in make invocation.
Sorry for the noise.

Re: TCP_NODELAY in tcp mode

[Willy Tarreau]

Hi Dmitry,

On Tue, Sep 08, 2015 at 05:25:33PM +0300, Dmitry Sivachenko wrote:
> 
> > On 30 ??. 2015 ??., at 22:29, Willy Tarreau  wrote:
> > 
> > On Fri, Aug 28, 2015 at 11:40:18AM +0200, Lukas Tribus wrote:
>  Ok, you may be hitting a bug. Can you provide haproxy -vv output?
>  
> >>> 
> >>> 
> >>> What do you mean? I get the following warning when trying to use this
> >>> option in tcp backend/frontend:
> >> 
> >> Yes I know (I didn't realize you are using tcp mode). I don't mean the
> >> warning is the bug, I mean the tcp mode is supposed to not cause any
> >> delays by default, if I'm not mistaken.
> > 
> > You're not mistaken, tcp_nodelay is unconditional in TCP mode and MSG_MORE
> > is not used there since we never know if more data follows. In fact there's
> > only one case where it can happen, it's when data wrap at the end of the
> > buffer and we want to send them together.
> > 
> 
> 
> Hello,
> 
> yes, you are right, the problem is not TCP_NODELAY.  I performed some testing:
> 
> Under low network load, passing TCP connection through haproxy involves 
> almost zero overhead.
> When load grows, at some point haproxy starts to slow things down.
> 
> In our testing scenario the application establishes long-lived TCP connection 
> to server and sends many small requests.
> Typical traffic at which adding haproxy in the middle causes measurable 
> slowdown is ~30MB/sec, ~100kpps.

This is not huge, it's smaller than what can be achieved in pure HTTP mode,
where I could achieve about 180k req/s end-to-end, which means at least 180kpps
in both directions on both sides, so 360kpps in each direction.

> haproxy process CPU usage is about 15-20%.

And the rest is for the system ?

Willy

Re: haproxy resolvers, DNS query not send / result NXDomain not expected

[Baptiste]

Hi Piba,

Finally, Willy fixed it in a different (and smarter) way:
http://git.haproxy.org/?p=haproxy.git;a=commit;h=07101d5a162a125232d992648a8598bfdeee3f3f

Baptiste

Re: HAProxy flow control

[PiBa-NL]

Op 8-9-2015 om 13:16 schreef ruoyu Zhang:

Hi,

I've read the HAProxy 1.5 Configuration Manual, and I find 
restrictions for connection numbers, session numbers, etc. However, I 
notice that there is not any restriction to control the flow rate as 
they focus more on the numbers rather than the flow rate. I know that 
iptables can realize this function but I this it will be more 
convenient if HAProxy can provide such kind of service. Will this be 
in the blueprint of HAProxy future version?


Best Regards,

Ruoyu Zhang

Hi,
Perhaps this is what your looking for?
"acl bruteforce_detection  sc2_http_req_rate gt 5"
As explained in: 
http://blog.haproxy.com/2013/04/26/wordpress-cms-brute-force-protection-with-haproxy/

Regards,
PiBa-NL

Re: SSL errors with HAProxy

[PiBa-NL]

Op 8-9-2015 om 14:17 schreef Brendan Kearney:

listen https 192.168.120.2:443
remove the address and port from the listen directive above, as 
currently it will bind port 443 twice, once with once without ssl 
offloading, leading to unpredictable results..

bind 192.168.120.2:443 ssl crt /etc/haproxy/www.pem
server www1 192.168.88.1:80
server www2 192.168.88.2:80