Re: Performing URL rewriting and re-resolving new URL using configured DNS

[francis Lavalliere]

Hello,

Haproxy is a great tool indeed. And as long as you are able to add some dynamic 
template generation of haproxy config based on your own business logic I 
believe you should be good.

You could leverage consul / consul-template and have the haproxy regenerates 
bases on your business logic.

We do use that combination and it works quite well for us.

Regards
Francis L.

Téléchargez Outlook pour Android


From: Yehuda Cohen 
Sent: Friday, September 11, 2020 11:32:51 AM
To: haproxy@formilux.org 
Subject: Performing URL rewriting and re-resolving new URL using configured DNS


Hi all,



Hoping you can help me with this query. I have a selection of micro-backends 
deployed per customer as containers on Amazon ECS. These customers each have a 
public hostname they use to access their respective micro-backend.



Currently we are using Amazon’s ALB to dispatch to these services, but each 
load balancer listener has a maximum of 100 rules that can dispatch traffic. As 
such we are managing multiple load balancers, and this something I’d rather not 
do. We are expecting to onboard thousands of customers and are in the process 
of automating the onboarding workflow.



One solution I thought of was to enable service discovery for each of these 
micro-backends to create an internal SRV record with a domain name that can be 
inferred from the public hostname. For example: 
https://customer-a.mypublicdomain.com to http://customer-a.myinternal.domain.



I’d then like to point each public hostname to a single reverse proxy 
(preferably configured with HAProxy). The reverse proxy would then forward the 
request from the public endpoint to the internal domain name by rewriting the 
Host header replacing mypublicdomain.com with myinternal.domain. It would then 
resolve the SRV record for myinternal.domain using a resolver DNS and forward 
the traffic to the internal micro-backend.



In this configuration, I’m looking to have HAProxy rewrite the url and then 
reresolve the rewritten url with a provided DNS rather than setting up a 
thousand backends and having to provide server configuration.



Is HAProxy a good tool for this use case? If so can anyone point me in a 
direction that I might use to configure this. So far every HAProxy 
configuration manual I’ve found that uses service discovery does so requires a 
backend block per service name. In this case micro-backends are added and 
deleted with very high frequency and I’d rather not modify HAProxy config files 
that frequently.



All the best,

Yehuda

Re: Bind socket not exists from time to time?

[francis Lavalliere]

So just to be clear, there is only 1 service, using a Master -> Worker
config.

The service is working since many hours (no doing any restart, simply
modification of the config and executing the hot-reload command)...

I'll try to dig deeper on myside also when I got a chance, I was mostly
wondering if anyone had that kind of issue (i know using sockets file isnt
much people are using).

- Francis L.

On Thu, Apr 11, 2019 at 12:19 PM Willy Tarreau  wrote:

> Hello Francis,
>
> On Thu, Apr 11, 2019 at 11:20:57AM -0400, francis Lavalliere wrote:
> > Hello,
> >
> > Anyone else would have seen this behavior before?
> > I am currently using HAProxy 1.9.3
> >
> > I have an "haproxy / consul template" that generate an haproxy-x.sock
> > file, from time to time when my config reloads, I find out after a few
> > minutes that the service doesn't work and actually the socket file doesnt
> > exists on the disk...
> >
> > Going to the HA Proxy server i can see  the haproxy has a valid
> > configuration, I even reload the worker threads, and the socket file
> still
> > does not exists on disk.
> >
> > ie:
> >
> > listen x
> >
> > bind /var/run/haproxy-x.sock accept-proxy
> >
> > bind-process 2
> >
> > I am wondering what could be the cause of this. I am assuming that the
> > haproxy template that I generate at some point create that listen, reload
> > the worker threads) then regenerate another configuration without that
> > socket, it then get removed... and then re-regenerate the config with the
> > listen and socket directive... but somehow the socket file doesnt get
> > reecreated.
> >
> > I might be wrong, as for now I have not much other possible ways to
> > reproduce easily as its happening randomly.
>
> I've not seen this but I can more or less imagine something that could
> cause this. Let's imagine that for whatever reason your service scripts
> start two haproxy processes in parallel, which end up like this :
>
>process1   process2
>   parse conf  parse conf
>   unlink("/var/run/..")
>   bind("/var/run/..")
>   unlink("/var/run/..")
>   bind("/var/run/..")
>   start other stuff   start other stuff
>   fail on something
>   unlink("/var/run/...")
>   exitcontinue to run
>
> In this case you can end up with process 2 running with process 1 having
> removed its socket while cleaning up on exit. It would randomly happen
> depending on the start order but that might be a possible explanation.
> Now maybe it's totally unrelated but I don't have any other idea for
> now.
>
> Regards,
> Willy
>

Bind socket not exists from time to time?

[francis Lavalliere]

Hello,

Anyone else would have seen this behavior before?
I am currently using HAProxy 1.9.3

I have an "haproxy / consul template" that generate an haproxy-x.sock
file, from time to time when my config reloads, I find out after a few
minutes that the service doesn't work and actually the socket file doesnt
exists on the disk...

Going to the HA Proxy server i can see  the haproxy has a valid
configuration, I even reload the worker threads, and the socket file still
does not exists on disk.

ie:

listen x

bind /var/run/haproxy-x.sock accept-proxy

bind-process 2

I am wondering what could be the cause of this. I am assuming that the
haproxy template that I generate at some point create that listen, reload
the worker threads) then regenerate another configuration without that
socket, it then get removed... and then re-regenerate the config with the
listen and socket directive... but somehow the socket file doesnt get
reecreated.

I might be wrong, as for now I have not much other possible ways to
reproduce easily as its happening randomly.

Re: invalid port numbers in 'show servers state' with SRV records

[francis Lavalliere]

Hello,

In server.c on line 2912


I believe the server state file is unsigned long and when loaded back its
being compared with USHRT_MAX

In my own development environment I had commented the following if
statement, as server state is fully controlled by the haproxy and
everything was working fine. But since this isn't the "Real" fix Ii assume
there will be needed of some kind of conversion to have a valid comparison.


```
if (port > USHRT_MAX) {
chunk_appendf(msg, ", invalid srv_port value '%s'", port_str);
port_str = NULL;
```

I

On Wed, Sep 19, 2018 at 12:32 PM Peter Erickson  wrote:

> Hello - I'm attempting to use HAProxy's service discovery (SRV records)
> along with `load-server-state-from-file` but the port numbers from `show
> servers state` are incorrect.
>
> Here is my example (HA-Proxy version 1.9-dev2 2018/09/12)
>
> DNS configuration:
>
> $ dig SRV _nginx._tcp.alpha.int
> ;; ANSWER SECTION:
> _nginx._tcp.alpha.int.  18  IN  SRV 1 1 32772
> 56b9c088-9bc5-4466-99af-ebebd3550791._nginx._tcp.alpha.int.
> _nginx._tcp.alpha.int.  18  IN  SRV 1 1 32773
> c6032796-8c7a-4d78-8934-d98900c18383._nginx._tcp.alpha.int.
>
> haroxy.cfg:
>
> global
> pidfile /var/run/haproxy.pid
> stats socket /haproxy-run/haproxy.sock
> server-state-file /haproxy-run/haproxy.state
>
> defaults
> modehttp
> timeout connect 1
> timeout client 10
> timeout server 10
> load-server-state-from-file global
>
> resolvers resolver0
> nameserver dns0 172.20.0.2:53
>
> frontend http
> bind *:80
> default_backend b1
>
> backend b1
> server-template www 2 _nginx._tcp.alpha.int check resolvers resolver0
>
> server state:
>
> $ echo "show servers state b1" | sudo socat haproxy-run/haproxy.sock stdio
> 1
> # be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state
> srv_uweight srv_iweight srv_time_since_last_change srv_check_status
> srv_check_result srv_check_health srv_check_state srv_agent_state
> bk_f_forced_id srv_f_forced_id srv_fqdn srv_port srvrecord
> 3 b1 1 www1 172.20.2.109 2 0 1 1 13 6 3 4 6 0 0 0
> 56b9c088-9bc5-4466-99af-ebebd3550791._nginx._tcp.alpha.int 4294934532
> _nginx._tcp.alpha.int
> 3 b1 2 www2 172.20.0.192 2 0 1 1 12 6 3 4 6 0 0 0
> c6032796-8c7a-4d78-8934-d98900c18383._nginx._tcp.alpha.int 4294934533
> _nginx._tcp.alpha.int
>
>
> As you can see, the port number in the SRV records are:
> 32772
> 32773
>
> However, the srv_port value from `show servers state` is:
> 4294934532
> 4294934533
>
> This causes the following error when haproxy attempts to read the state
> file:
> [WARNING] 261/162616 (1) : server-state application failed for server
> 'b1/www1', invalid srv_port value '4294934532'
> [WARNING] 261/162616 (1) : server-state application failed for server
> 'b1/www2', invalid srv_port value '4294934533'
>
> Can anyone help me out?
>
> Thanks,
> Pete
>

Re: Observations about reloads and DNS SRV records

[francis Lavalliere]

This is related to the discourse thread (And also discuss the issues in ML
instead of discourse).

https://discourse.haproxy.org/t/config-reload-with-dynamic-service-discovery-via-dns/2625/10


Here are the findings from my ends:



1 - State file / Port range invalid conversion

In the haproxy state file the Port is written as an unsigned long value..

ie: Port 32777 is actually written 4294934537

112 defaultback_failsaife 1 varnish1 10.100.20.78 2 0 1 1 203910 15 3 4 6 0
0 0 ip-10-100-20-78.node.aws-us-east-1.consul 4294934537 _tcp_.varnish
.service.consul


To allow the seamless reload, I had to comment the following lines in the
src/server.c


if (port > USHRT_MAX) {
  chunk_appendf(msg, “, invalid srv_port value ‘%s’”, port_str);
  port_str = NULL;
}


2 - Backend responds with 503 only after reload

In order to make it functional,

in the patch  i had to comment the following lines :

/*
// prepare DNS resolution for this server (but aint this has already been
done by the server-template function?)
res = srv_prepare_for_resolution(srv, fqdn);
if (res == -1) {
ha_alert(“could not allocate memory for DNS REsolution for server …
‘%s’\n”, srv->id);
chunk_appendf(msg, “, can’t allocate memory for DNS resolution for server
‘%s’”, srv->id);
HA_SPIN_UNLOCK(SERVER_LOCK, >lock);
goto out;
}
*/