RE: Http HealthCheck Issue

[UPPALAPATI, PRAVEEN]

Hi Alex,

If I have one vhost representing all the nexus host's how can haproxy identify 
which server is down ?

I guess health check is to determine which server is healthy right if a vhost 
masks all the servers in the backend list how would haproxy divert the traffic?

Please advise.

Thanks,
Praveen.

-Original Message-
From: Aleksandar Lazic [mailto:al-hapr...@none.at] 
Sent: Thursday, December 20, 2018 2:34 AM
To: UPPALAPATI, PRAVEEN ; haproxy 
Subject: Re: Http HealthCheck Issue

Hi Praveen.

Please keep the list in the loop, thanks.

Am 20.12.2018 um 07:00 schrieb UPPALAPATI, PRAVEEN:
> Hi Alek,
> 
> Now I am totally confused:
> 
> When I say :
> 
> 
> backend bk_8093_read
> balancesource
> http-response set-header X-Server %s
> option log-health-checks
> option httpchk GET 
> /nexus/repository/rawcentral/com.att.swm.attpublic/healthcheck.txt 
> HTTP/1.1\r\nHost:\ server1.com:8093\r\nAuthorization:\ Basic\ ...
> server primary8093r server1.com:8093 check verify none
> server backUp08093r server2.com:8093 check backup verify none 
> server backUp18093r server3.com:8093 check backup verify none
> 
> Here server1.com,server2.com and physical server hostnames.

That's the issue. You should have a vhost name which is for all servers the 
same.

> When I define HOST header which server should I define I was expecting 
> haproxy will formulate that to 
> 
> https://urldefense.proofpoint.com/v2/url?u=http-3A__server1.com-3A8093_nexus_repository_rawcentral_com.att.swm.attpublic_healthcheck.txt&d=DwIFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=V0kSKiLhQKpOQLIjj3-g9Q&m=OZ-O9xFFKCfaCy769FVtGWPRgXm2eydV92WYJPam8Yg&s=--6orxuqJAZSK_-qumJjuZEhy1Iru7mkgPPJvYpw4RQ&e=
> https://urldefense.proofpoint.com/v2/url?u=http-3A__server2.com-3A8093_nexus_repository_rawcentral_com.att.swm.attpublic_healthcheck.txt&d=DwIFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=V0kSKiLhQKpOQLIjj3-g9Q&m=OZ-O9xFFKCfaCy769FVtGWPRgXm2eydV92WYJPam8Yg&s=eJkYBH7JAMAIU1ZIHXCFbOs3RLA0OtMHp1ky_rN-d7s&e=
> https://urldefense.proofpoint.com/v2/url?u=http-3A__server3.com-3A8093_nexus_repository_rawcentral_com.att.swm.attpublic_healthcheck.txt&d=DwIFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=V0kSKiLhQKpOQLIjj3-g9Q&m=OZ-O9xFFKCfaCy769FVtGWPRgXm2eydV92WYJPam8Yg&s=sUUE7fa4pe0lN8iFaAuznj-m8sgwQS2X2xeeva-yCEM&e=
> 
> to monitor which servers are live right , so how could I dynamically populate 
> the HOST?

I have written the following:

> I assume that nexus have a general URL and not srv1,srv2, ...

This is also mentioned in the config example.

https://urldefense.proofpoint.com/v2/url?u=https-3A__help.sonatype.com_repomanager2_installing-2Dand-2Drunning_running-2Dbehind-2Da-2Dreverse-2Dproxy&d=DwIFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=V0kSKiLhQKpOQLIjj3-g9Q&m=OZ-O9xFFKCfaCy769FVtGWPRgXm2eydV92WYJPam8Yg&s=vJtNj9Bd4EYr_kdknJiGcOOfWnV6Mgna31-JAbiKwq4&e=

This means on the nexus should be a generic hostname which is for all servers
the same.

Maybe you can share the nexus config, nexus setup and the nexus version, as it's
normally not a big deal to setup a vhost.

Regards
Aleks


PS: What's this urldefense.proofpoint.com crap 8-O

> Please advice.
> 
> Thanks,
> Praveen.
> 
> 
> 
> -Original Message-
> From: Aleksandar Lazic [mailto:al-hapr...@none.at] 
> Sent: Wednesday, December 19, 2018 3:25 PM
> To: UPPALAPATI, PRAVEEN 
> Cc: Jonathan Matthews ; Cyril Bonté 
> ; haproxy@formilux.org
> Subject: Re: Http HealthCheck Issue
> 
> Am 19.12.2018 um 21:04 schrieb UPPALAPATI, PRAVEEN:
>> Ok then do I need to add the haproxy server?
> 
> I suggest to use a `curl -v
> /nexus/v1/repository/rawcentral/com.att.swm.attpublic/healthcheck.txt`
> and see how curl make the request.
> 
> I assume that nexus have a general URL and not srv1,srv2, ...
> For example.
> 
> ###
> curl -vo /dev/null 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.haproxy.org&d=DwIFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=V0kSKiLhQKpOQLIjj3-g9Q&m=CaQ1GDp8D6XzObaEV3Ad9IQ3Q1TwhAAYhFQ24IgwP68&s=53v5RKBVFzzKyU7JGcd8i6eBlGyIfSavQBRkoYcXZm8&e=
> 
> * Rebuilt URL to: 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.haproxy.org_&d=DwIFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=V0kSKiLhQKpOQLIjj3-g9Q&m=CaQ1GDp8D6XzObaEV3Ad9IQ3Q1TwhAAYhFQ24IgwP68&s=k3KIXBm21aPgFFyUUyjSUxggMPVWIqkWYwdaWeDZ6NI&e=
>   % Total% Received % Xferd  Average Speed   TimeTime Time  
> Current
>  Dload  Upload   Total   SpentLeft  Speed
>   0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
> 0*
>   Trying 51.15.8.218...
> * Connected to 
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.haproxy.org&d=DwIFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=V0kSKiLhQKpOQLIjj3-g9Q&m=CaQ1GDp8D6XzObaEV3Ad9IQ3Q1TwhAAYhFQ24IgwP68&s=WMHYWYJP4ycNXvPYov7PnJdkQB26fgfXm_ByW2BCM8g&e=
>  (51.15.8.218) port 443 (#0)
> * found 148 certificates in /etc/ssl/certs/ca-certificates.crt
> * found 599 certificates in /etc/ssl/certs
> * ALPN, offering http/1.1
> * SSL connection using TLS1.2 / ECDHE_RSA_

Re: haproxy AIX 7.1.0.0 compile issues

[Aleksandar Lazic]

Hi Patrick.

Am 26-12-2018 22:26, schrieb Overbey, Patrick (Sioux Falls):


Hello,

First off, I want to say thank you for your hard work on haproxy. It is 
a very fine piece of software.
I recently ran into a bug compiling haproxy 1.9+ on AIX 7.1 
7100-00-03-1115 using gmake 4.2 and gcc 8.1.0. I had previously had 1.8 
compiling using this same setup with a few minor code changes to 
variable names in vars.c and connection.c. I made those same changes in 
version 1.9, but have now ran into a compile issue that I cannot get 
around due to initcall.h being new in 1.9.


Please can you tell us which `minor code changes` was necessary to be 
able to compile on AIX 7.1.



Here are the compile errors I am seeing.

ld: 0711-345 Use the -bloadmap or -bnoquiet option to obtain more 
information.


What do you get when you add `-bnoquiet` to the LDFLAGS?
Please can you share the compile output with `V=1`  activated.


ld: 0711-317 ERROR: Undefined symbol: __start_init_STG_PREPARE

ld: 0711-317 ERROR: Undefined symbol: __stop_init_STG_PREPARE

ld: 0711-317 ERROR: Undefined symbol: __start_init_STG_LOCK

ld: 0711-317 ERROR: Undefined symbol: __stop_init_STG_LOCK

ld: 0711-317 ERROR: Undefined symbol: __start_init_STG_ALLOC

ld: 0711-317 ERROR: Undefined symbol: __stop_init_STG_ALLOC

ld: 0711-317 ERROR: Undefined symbol: __start_init_STG_POOL

ld: 0711-317 ERROR: Undefined symbol: __stop_init_STG_POOL

ld: 0711-317 ERROR: Undefined symbol: __start_init_STG_REGISTER

ld: 0711-317 ERROR: Undefined symbol: __stop_init_STG_REGISTER

ld: 0711-317 ERROR: Undefined symbol: __start_init_STG_INIT

ld: 0711-317 ERROR: Undefined symbol: __stop_init_STG_INIT

Would anyone have suggestions for how to fix this?

Also, as a note I am able to compile haproxy 1.5 out of the box, but 
starting with version 1.6 is where I run into compile errors. Is there 
support for these compile bugs or am I on my own?


Please can you be more precise which compile errors you got, thanks.


Thanks for any help you can offer.

PATRICK OVERBEY

Software Development Engineer Staff

Product Development/Bank Solutions

Office: 605-362-1260  x7290

FISERV
JOIN US @ FORUM 2019
Fiserv | Join Our Team | Twitter | LinkedIn | Facebook
FORTUNE Magazine WORLD'S MOST ADMIRED COMPANIES® 2014 | 2015 | 2016 | 
2017 | 2018
(c) 2018 Fiserv Inc. or its affiliates. Fiserv is a registered 
trademark of Fiserv Inc. Privacy Policy

Re: [PATCH] BUG/MINOR: lb: fix redispatch for hash based lb-algo's

[Lukas Tribus]

Hello,


On Wed, 26 Dec 2018 at 23:04, Lukas Tribus  wrote:
>
> redispatch never worked for hash based alghoritms, as the code
> (BE_LB_LKUP_CHTREE -> chash_get_next_server()) would only have been
> called for BE_LB_KIND_RR, which doesn't make sense. Fix this by also
> going down this code path when the BE_LB_KIND is BE_LB_KIND_HI.
>
> Reported by Oskar Stenman on discourse:
> https://discourse.haproxy.org/t/balance-uri-consistent-hashing-redispatch-3-not-redispatching/3344
>
> Can be backported to 1.6.

Actually my intention was to set this as RFC, with the following
comments, but git send-mail was faster:

I'm not 100% sure whether "option redispatch" was only intended to
break cookie persistence, but not other lb algorithms. Docs are
ambiguous about this.

However, since option redispatch is configurable per backend, it
doesn't make a lot of sense to exclude hash based algorithms from
redispatch functionality.

Also, the fact the code is already there, it's just never called
(s->be->lbprm.algo is never BOTH BE_LB_KIND_RR and BE_LB_LKUP_CHTREE)
makes me think that this was just an oversight.

Backporting this could change behavior as redispatch suddenly begins
to work in such a situation; however this is expected and a correct
configuration fixes it immediately.



Regards,
Lukas

[PATCH] BUG/MINOR: lb: fix redispatch for hash based lb-algo's

[Lukas Tribus]

redispatch never worked for hash based alghoritms, as the code
(BE_LB_LKUP_CHTREE -> chash_get_next_server()) would only have been
called for BE_LB_KIND_RR, which doesn't make sense. Fix this by also
going down this code path when the BE_LB_KIND is BE_LB_KIND_HI.

Reported by Oskar Stenman on discourse:
https://discourse.haproxy.org/t/balance-uri-consistent-hashing-redispatch-3-not-redispatching/3344

Can be backported to 1.6.
---
 src/backend.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/backend.c b/src/backend.c
index bc38c57..a402a86 100644
--- a/src/backend.c
+++ b/src/backend.c
@@ -637,7 +637,7 @@ int assign_server(struct stream *s)
 
case BE_LB_LKUP_CHTREE:
case BE_LB_LKUP_MAP:
-   if ((s->be->lbprm.algo & BE_LB_KIND) == BE_LB_KIND_RR) {
+   if ((s->be->lbprm.algo & BE_LB_KIND) == BE_LB_KIND_RR 
|| (s->be->lbprm.algo & BE_LB_KIND) == BE_LB_KIND_HI) {
if ((s->be->lbprm.algo & BE_LB_PARM) == 
BE_LB_RR_RANDOM)
srv = get_server_rnd(s);
else if (s->be->lbprm.algo & BE_LB_LKUP_CHTREE)
-- 
2.7.4

haproxy AIX 7.1.0.0 compile issues

[Overbey, Patrick (Sioux Falls)]

Hello,
First off, I want to say thank you for your hard work on haproxy. It is a very 
fine piece of software.
I recently ran into a bug compiling haproxy 1.9+ on AIX 7.1 7100-00-03-1115 
using gmake 4.2 and gcc 8.1.0. I had previously had 1.8 compiling using this 
same setup with a few minor code changes to variable names in vars.c and 
connection.c. I made those same changes in version 1.9, but have now ran into a 
compile issue that I cannot get around due to initcall.h being new in 1.9.

Here are the compile errors I am seeing.

ld: 0711-345 Use the -bloadmap or -bnoquiet option to obtain more information.
ld: 0711-317 ERROR: Undefined symbol: __start_init_STG_PREPARE
ld: 0711-317 ERROR: Undefined symbol: __stop_init_STG_PREPARE
ld: 0711-317 ERROR: Undefined symbol: __start_init_STG_LOCK
ld: 0711-317 ERROR: Undefined symbol: __stop_init_STG_LOCK
ld: 0711-317 ERROR: Undefined symbol: __start_init_STG_ALLOC
ld: 0711-317 ERROR: Undefined symbol: __stop_init_STG_ALLOC
ld: 0711-317 ERROR: Undefined symbol: __start_init_STG_POOL
ld: 0711-317 ERROR: Undefined symbol: __stop_init_STG_POOL
ld: 0711-317 ERROR: Undefined symbol: __start_init_STG_REGISTER
ld: 0711-317 ERROR: Undefined symbol: __stop_init_STG_REGISTER
ld: 0711-317 ERROR: Undefined symbol: __start_init_STG_INIT
ld: 0711-317 ERROR: Undefined symbol: __stop_init_STG_INIT

Would anyone have suggestions for how to fix this?

Also, as a note I am able to compile haproxy 1.5 out of the box, but starting 
with version 1.6 is where I run into compile errors. Is there support for these 
compile bugs or am I on my own?

Thanks for any help you can offer.

Patrick Overbey
Software Development Engineer Staff
Product Development/Bank Solutions
Office: 605-362-1260  x7290

Fiserv
Join us @ Forum 
2019
Fiserv
 | Join Our 
Team
 | 
Twitter
 | 
LinkedIn
 | 
Facebook
FORTUNE Magazine World's Most Admired Companies(r) 2014 | 2015 | 2016 | 2017 | 
2018
(c) 2018 Fiserv Inc. or its affiliates. Fiserv is a registered trademark of 
Fiserv Inc. Privacy Policy

RE: Send-proxy not modifying some traffic with proxy ip/port details instead retaining same client ip port

[Mohandass, Roobesh]

Hello Lukas,

Sure, we will try the attached patch work and share the feedback to you/team.

Thanks for help.

Kind regards,
Roobesh G M

-Original Message-
From: lu...@ltri.eu  
Sent: Wednesday, December 26, 2018 6:30 PM
To: Mohandass, Roobesh 
Cc: haproxy 
Subject: Re: Send-proxy not modifying some traffic with proxy ip/port details 
instead retaining same client ip port

This email originated from outside of the organization. Do not click links or 
open attachments unless you recognize the sender and know the content is safe.

Hello Roobesh,


On Wed, 26 Dec 2018 at 11:49, Mohandass, Roobesh  
wrote:
> RGM: This is reproducible anywhere production/lab but when we see this 
> behavior is a questions as I said out of so many large number of 
> requests only for some we will observe this behavior (but can be caught very 
> quickly in less than 30mins span of time).

Thanks for clarifying.

I don't have the time to reproduce this today, but could you try the attached 
patch please?

It changes the logic around those calls, uses the getsockopt/SO_ORIGINAL_DST 
first, checks the return value and then calls getsockname(). Previously we did 
not check the return value of the getsockopt/SO_ORIGINAL_DST. My hope is that 
that call returns an error when it returns wrong data - if that's the case, the 
patch should fix the issue without sacrificing other haproxy features.

If that doesn't fix the problem we probably have to talk to the 
kernel/netfilter folks.


Thanks,
Lukas


1.8-SO_ORIGINAL_DST-change.diff
Description: 1.8-SO_ORIGINAL_DST-change.diff

Re: Send-proxy not modifying some traffic with proxy ip/port details instead retaining same client ip port

[Lukas Tribus]

Hello Roobesh,


On Wed, 26 Dec 2018 at 11:49, Mohandass, Roobesh
 wrote:
> RGM: This is reproducible anywhere production/lab but when we see this 
> behavior is a questions
> as I said out of so many large number of requests only for some we will 
> observe this behavior
> (but can be caught very quickly in less than 30mins span of time).

Thanks for clarifying.

I don't have the time to reproduce this today, but could you try the
attached patch please?

It changes the logic around those calls, uses the
getsockopt/SO_ORIGINAL_DST first, checks the return value and then
calls getsockname(). Previously we did not check the return value of
the getsockopt/SO_ORIGINAL_DST. My hope is that that call returns an
error when it returns wrong data - if that's the case, the patch
should fix the issue without sacrificing other haproxy features.

If that doesn't fix the problem we probably have to talk to the
kernel/netfilter folks.


Thanks,
Lukas


1.8-SO_ORIGINAL_DST-change.diff
Description: Binary data

RE: Send-proxy not modifying some traffic with proxy ip/port details instead retaining same client ip port

[Mohandass, Roobesh]

Hello Lukas,



What I would ask you is to clarify how this can be reproduced? Do you only see 
this in production, or can you reproduce this also in a lab?

If so, how do you reproduce it?



RGM: This is reproducible anywhere production/lab but when we see this behavior 
is a questions as I said out of so many large number of requests only for some 
we will observe this behavior(but can be caught very quickly in less than 
30mins span of time).
Initiate some larger numbers of requests (with some basic setup frontend with 
backend using send proxy and some backend servers) and perform an easy way of 
catching this on first hand using ' tcpdump -i  -nn -A net 
 and port  | grep PROXY | cut -c9- | awk '/^PROXY/ && $4 
!~ /^10\.10\./ && $4 !~ /^10\.11\./ && $4 !~ /^10\.12\./ {print $0 }'  (adjust 
the awk command according to your frontend/backend network) so this command 
will only result when data is wrong for destination IP (like below),
Below is just an illustration,
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bond0, link-type EN10MB (Ethernet), capture size 262144 bytes
PROXY TCP4 100.100.100.100 100.100.100.100 41135 41135
PROXY TCP4 100.100.100.100 100.100.100.100 11547 11547



You can try if you have your test setup , if not I can show you live in my 
staging if you are open for a webex call.



Thanks very much for your help and support.



-Roobesh G M



-Original Message-
From: lu...@ltri.eu 
Sent: Wednesday, December 26, 2018 3:09 PM
To: Mohandass, Roobesh ; haproxy 

Subject: Re: Send-proxy not modifying some traffic with proxy ip/port details 
instead retaining same client ip port



This email originated from outside of the organization. Do not click links or 
open attachments unless you recognize the sender and know the content is safe.



Hello Roobesh,



On Wed, 26 Dec 2018 at 08:31, Mohandass, Roobesh 
mailto:roobesh_mohand...@mcafee.com>> wrote:

>

> Hello,

>

>

>

> We are using haproxy version 1.8.14-1 in a docker container running

> ubuntu 14.04 / kernel: 4.15.0-39-generic (Base host where container is

> running 18.04 / kernel 4.15.0-39-generic)

>

> getsockopt(fd, SOL_IP, SO_ORIGINAL_DST, sa, &salen) is in fact sometimes 
> returning the source IP instead the destination IP.

>

> Using getsockname() instead looks like solving the issue.

>

>

>

> https://stackoverflow.com/questions/11417187/getsockopt-so-original-ds

> t-occasionally-returns-client-address

>

> For example: Out of 6569124 requests , 4 requests were wrong

> 0.60891 %

>

>

>

> Can we file this as bug ? Get the fixed version in next release ?

>

> Also getsockopt(fd, SOL_IP, SO_ORIGINAL_DST, sa, &salen) is used only for 
> send-proxy feature (for connected proxy IP) or any other feature in haproxy 
> also using this ?



For the record, the initial thread with you is on discourse:

https://discourse.haproxy.org/t/send-proxy-not-modifying-some-traffic-with-proxy-ip-port-details/3336



Martin Zielinski mentioned the hint that SO_ORIGINAL_DST sometimes fails and 
omitting that call (returning the result of getsockname()) works around the 
issue. You confirmed this by compiling with USE_TPROXY= which disables the code 
path in question (SO_ORIGINAL_DST).





SO_ORIGINAL_DST is certainly a feature that is required when haproxy binds to 
redirected sockets, otherwise we don't know the correct destination IP, so I 
don't think we can simply drop the call. Looking some further in into the root 
cause is what is required to get a better understanding of this.





What I would ask you is to clarify how this can be reproduced? Do you only see 
this in production, or can you reproduce this also in a lab?

If so, how do you reproduce it?







Thanks,

Lukas

Re: Send-proxy not modifying some traffic with proxy ip/port details instead retaining same client ip port

[Lukas Tribus]

Hello Roobesh,

On Wed, 26 Dec 2018 at 08:31, Mohandass, Roobesh
 wrote:
>
> Hello,
>
>
>
> We are using haproxy version 1.8.14-1 in a docker container running ubuntu 
> 14.04 / kernel: 4.15.0-39-generic (Base host where container is running 18.04 
> / kernel 4.15.0-39-generic)
>
> getsockopt(fd, SOL_IP, SO_ORIGINAL_DST, sa, &salen) is in fact sometimes 
> returning the source IP instead the destination IP.
>
> Using getsockname() instead looks like solving the issue.
>
>
>
> https://stackoverflow.com/questions/11417187/getsockopt-so-original-dst-occasionally-returns-client-address
>
> For example: Out of 6569124 requests , 4 requests were wrong 0.60891 %
>
>
>
> Can we file this as bug ? Get the fixed version in next release ?
>
> Also getsockopt(fd, SOL_IP, SO_ORIGINAL_DST, sa, &salen) is used only for 
> send-proxy feature (for connected proxy IP) or any other feature in haproxy 
> also using this ?

For the record, the initial thread with you is on discourse:
https://discourse.haproxy.org/t/send-proxy-not-modifying-some-traffic-with-proxy-ip-port-details/3336

Martin Zielinski mentioned the hint that SO_ORIGINAL_DST sometimes
fails and omitting that call (returning the result of getsockname())
works around the issue. You confirmed this by compiling with
USE_TPROXY= which disables the code path in question
(SO_ORIGINAL_DST).


SO_ORIGINAL_DST is certainly a feature that is required when haproxy
binds to redirected sockets, otherwise we don't know the correct
destination IP, so I don't think we can simply drop the call. Looking
some further in into the root cause is what is required to get a
better understanding of this.


What I would ask you is to clarify how this can be reproduced? Do you
only see this in production, or can you reproduce this also in a lab?
If so, how do you reproduce it?



Thanks,
Lukas