Re: [PR/FEATURE] support for virtual hosts / Host header per server

[Willy Tarreau]

Hello Romain,

On Tue, Oct 01, 2019 at 12:08:03PM +, Morotti, Romain D wrote:
> What is the status on this? 

Sorry, but it took some time to work on other priorities, and to be
honest, the subject looked scary enough to deserve enough time to
study it. Sadly, if I can say, the subject was pretty descriptive of
what it does, and this is fundamentally wrong.

So just to summarize for those who haven't had a look at the patch,
what this patch does is to replace the host header in requests sent to
a server with one specified for this server, resulting in each server
within the same farm to run on a different vhost. This goes back to
the errors that plagued many hosting infrastructures in the late 90s
and early 2000s where redirects, hosts in pages etc were wrong because
the application was called with an internal name instead of the correct
one. Moreover this used to prevent servers from being shared between
multiple hosts since the host header was containing rubish. For
reference, Apache merged the ProxyPreserveHost in 2.0.31 in 2002 to put
an end to that madness. But here you're escalating this to a new level
and involving it in load balancing, 15 years after everyone managed to
fix that early mistake and bury it deeply forever. So in short for a
request sent to foo.bar.com, you'll randomly forward it as being for
foo1.bar.com, bar.foo2.com, www.example.org etc... all in the same farm!
It's as if when launching my browser on Amazon I was presented eBay,
Amazon and Aliexpress in a round robin fashion. I'm exagerating but it
really is the same principle.

It is critically important that Host headers are not tampered with
anymore, especially in a load balanced setup where all nodes must
absolutely receive the exact same requests. Nowadays they're used
way more than in the early 2000, you have them in Set-Cookie, 
Access-Control-Allow-Origin header with CORS, Location for redirects
(obviously), logs, statistics. It makes absolutely no sense to deploy
multiple servers in a same farm on different host names for the same
service. And no, migrating from an old Apache 1.3 config relying on
ProxyPass is not a valid excuse in 2019 for reintroducing such 
architectural flaws!

Such mistakes are still occasionally encountered in field, but extremely
rarely. Usually it's after their author, and single defender, quits the
company and their substitute discovers the mess and says "Houston, we
have a problem". But fortunately while I've met it very few times with
reverse proxies over the last decade, I never met it at all within a
load balancing farm in 18 years!

I'm seeing this as the outcome of someone having shoot himself in the
foot, then aiming at the knee in order not to feel the foot's pain
anymore. But here I don't want to engage every users' knee in front
of this gun.

For the part concerning the health checks however your point is totally
valid and I absolutely hate the "\r\nHost:foo" hack that we're relying on
from time to time. I'd like to see checks reworked so that we can pass
various headers etc. I'm not fond of having just a vhost directive to be
used by checks because 1) it's confusing and 2) we're just hiding the dust
under the carpet. We do need really improved checks and dealing with Host
only is not sufficient. Some users need to pass cookies, auth headers,
user-agent, or x-forwarded-for headers for example, and such configs are
really really dirty. Usually they switch to tcp-checks but then they cannot
use fancy features like disable-on-404.

> The lack of this functionality is a blocker to use HAProxy with kubernetes.

I'd say that many existing haproxy users in kubernetes would probably
disagree with this bold claim. Saying that you're facing difficulties
and are seeking for configuration help would possibly be more accurate.

In my opinion at this point it's preferable that you clearly describe
the problem you're facing so that it can be split in 3 parts :

  - architecture issues
  - configuration issues
  - haproxy limitations

I really suspect that you're facing a bit of each, and that for example
you're having an issue with your checks being unusable or unreliable on
the *real* application, forcing you to check a dummy one running on the
server itself, which does expect to be called with its own name and that
since haproxy uses the same check requests for all hosts you're facing
difficulties running the dummy checks. But maybe fixing the application
so that it can really be checked would be the best solution, maybe
configuring the dummy check application with a proper host would be the
second best solution, maybe using HTTP/1.0 checks to get rid of Host
(as many people do notably on static servers to end up on the default
vhost) would work fine, or maybe we need to perform minimal changes to
the checks to deal with a specific case.

Among the things I've been thinking about in the past regarding short
term improvements to the checks that could remain compatible with a
whole rewrite, I'd 

RE: [PR/FEATURE] support for virtual hosts / Host header per server

[Sayar, Guy H]

Hi Baptiste

The use case we are looking at specifically is to host a web service on 
kubernetes alongside a the same web service on a plain node, both being served 
by the same ha proxy backend.

The health check for the kubernetes servers fails due to the host header issue, 
so we end up with inability to route traffic to the kubernetes based servers, 
or disable health check both of which are not desired.

Regards
Guy


From: Baptiste [mailto:bed...@gmail.com]
Sent: 01 October 2019 14:08
To: Morotti, Romain D (CIB Tech, GBR) 
mailto:romain.d.moro...@jpmorgan.com>>
Cc: haproxy@formilux.org; Sayar, Guy H (CIB Tech, 
GBR) mailto:guy.h.sa...@jpmorgan.com>>
Subject: Re: [PR/FEATURE] support for virtual hosts / Host header per server

Hi Romain,

Can you tell us (or me individually) why you can't use HAProxy with Kubernetes 
because of this?
I am interested by the use case.

Baptiste


On Tue, Oct 1, 2019 at 2:10 PM Morotti, Romain D 
mailto:romain.d.moro...@jpmorgan.com>> wrote:
What is the status on this?

The lack of this functionality is a blocker to use HAProxy with kubernetes.

-Original Message-
From: Willy Tarreau [mailto:w...@1wt.eu]
Sent: 01 August 2019 04:59
To: Morotti, Romain D 
mailto:romain.d.moro...@jpmorgan.com>>
Cc: haproxy@formilux.org
Subject: Re: [PR/FEATURE] support for virtual hosts / Host header per server

Hello Romain,

On Wed, Jul 31, 2019 at 04:02:04PM +, Morotti, Romain D wrote:
> Hello,
>
> Didn't get any reply here. Is anybody reviewing this mailing list?

Sorry about this but I simply think that most developers are busy chasing 
complex bugs and since it's the holiday period it's more difficult to find time 
to review patches.

Regards,
Willy


This message is confidential and subject to terms at: 
https://www.jpmorgan.com/emaildisclaimer including on confidential, privileged 
or legal entity information, viruses and monitoring of electronic messages. If 
you are not the intended recipient, please delete this message and notify the 
sender immediately. Any unauthorized use is strictly prohibited.

This message is confidential and subject to terms at: 
https://www.jpmorgan.com/emaildisclaimer including on confidential, privileged 
or legal entity information, viruses and monitoring of electronic messages. If 
you are not the intended recipient, please delete this message and notify the 
sender immediately. Any unauthorized use is strictly prohibited.

Re: Use haproxy behind Squid

[Aleksandar Lazic]

Am 02.10.19 um 13:10 schrieb Akhnin Nikita:
> Hey there!
> 
> Is it possible to use Haproxy behind HTTP proxy like Squid to proxy incoming
> requests to the Internet through it? It will be awesome if someone will share
> the configuration example.

Do you mean such a flow?

Internet -> squid -> haproxy -> Client

This statement confuses me a little bit.

> to proxy incoming requests to the Internet

>From which point of view is incomming and outgoing?

Regards
Aleks

Use haproxy behind Squid

[Akhnin Nikita]

Hey there!

Is it possible to use Haproxy behind HTTP proxy like Squid to proxy incoming 
requests to the Internet through it? It will be awesome if someone will share 
the configuration example.

healthchecks (to uwsgi) possible regression 1.9.8 -> 1.9.9

[Jarno Huuskonen]

Hello,

I was testing haproxy -> uwsgi(alert.io) and noticed a possible regression
with healthchecks(httpchk).
With 1.9.9 uwsgi logs:
[uwsgi-http key: host.name.fi client_addr: 127.0.0.1 client_port: 45715] 
hr_read(): Connection reset by peer [plugins/http/http.c line 917]

health checks work
(option httpchk GET /_ HTTP/1.1\r\nHost:\ host.name.fi\r\nUser-Agent:\ haproxy)
but uwsgi logs the hr_read() warning/error.

I bisected 1.9.9 and this commit is probably the commit that changes
behaviour between 1.9.8 and 1.9.9:
5d0cb90eb78f869e8801b34eddfdfd5dd8360e71 is the first bad commit
commit 5d0cb90eb78f869e8801b34eddfdfd5dd8360e71
Author: Olivier Houchard 
Date:   Fri Jun 14 15:26:06 2019 +0200

BUG/MEDIUM: connections: Don't call shutdown() if we want to disable linger.

In conn_sock_shutw(), avoid calling shutdown() if linger_risk is set. Not
doing so will result in getting sockets in TIME_WAIT for some time.
This is particularly observable with health checks.

This should be backported to 1.9.

(cherry picked from commit fe4abe62c7c5206dff1802f42d17014e198b9141)
Signed-off-by: Christopher Faulet 

Also 1.9.11, 2.0.7 and 2.1-dev2 has the same problem with uwsgi hr_read().
If I revert commits 6c7e96a3e1abb331e414d1aabb45d9fedb0254c2 and
fe4abe62c7c5206dff1802f42d17014e198b9141 from 2.1-dev2 then the uwsgi hr_read()
disappears.

If this seems worth digging into I can get packet captures or strace.
(I'm testing this is on rhel8 vm with 4.18.0-80.11.1.el8_0.x86_64 kernel).

This is fairly minimal config for testing:
frontend FE_alerta
bind ipv4@:8443 name alertav4ssl ssl crt /etc/haproxy/ssl/crtname.pem 
alpn h2,http/1.1

modehttp
option  dontlognull
option  http-ignore-probes  # ignore "pre-connect" requests
timeout http-request8s

capture request header Host len 40

option contstats
option forwardfor   except 127.0.0.0/8

# remove incoming X-Forwarded-For headers
http-request set-header X-Forwarded-Proto https

default_backend BE_alertaapi

#
# Alerta uwsgi backend
#
backend BE_alertaapi
option httpchk GET /_ HTTP/1.1\r\nHost:\ demo3.uef.fi\r\nUser-Agent:\ 
UEFHaproxy
http-check expect string OK
http-check disable-on-404

retries 2
option  redispatch
option  prefer-last-server
balance roundrobin

timeout connect 4500ms
timeout server  30s
timeout queue   4s
timeout check   5s

# uwsgi alerta app expects /alerts (not /api/alerts), strip /api
#http-request replace-uri ^/api/?(.*) /\1

# inter fast for uwsgi hr_read() testing
default-server inter 6s downinter 25s rise 2
server alertaapi1 127.0.0.1:8080 id 1 check

-Jarno

-- 
Jarno Huuskonen