Re: DNS resolution problem since 1.8.14

2018-12-25 Thread Willy Tarreau 
Hi Patrick,

On Mon, Dec 24, 2018 at 10:02:17AM +0100, Patrick Valsecchi wrote:
> Hi Jonathan,
> 
> I've build the 1.8.16 image myself and the problem is indeed fixed. Any plan
> of including that fix in a 1.9.1 release?

That's indeed planed, but as usual with any initial dot-zero release we've
found a number of small issues that deserve being addressed so it's never
the right moment to emit dot-one :-)

I'll try to do one this week.

Cheers,
Willy

Re: DNS resolution problem since 1.8.14

2018-12-24 Thread Patrick Valsecchi 

Hi Jonathan,

I've build the 1.8.16 image myself and the problem is indeed fixed. Any 
plan of including that fix in a 1.9.1 release?


Thanks.

On 23.12.18 18:20, Jonathan Matthews wrote:

Hey Patrick,

Have you looked at the fixes in 1.8.16? They sound kinda-sorta related 
to your problem ...


J

On Sun, 23 Dec 2018 at 16:17, Patrick Valsecchi > wrote:


I did a tcpdump. My config is modified to point to a local
container (www) in a docker compose (I'm trying to simplify my
setup). You can see the DNS answers correctly:

16:06:00.181533 IP (tos 0x0, ttl 64, id 63816, offset 0, flags
[DF], proto UDP (17), length 68)
    127.0.0.11.53 > localhost.40994: 63037 1/0/0 www. A
172.20.0.17 (40)

Could it be related to that?

https://github.com/haproxy/haproxy/commit/8d4e7dc880d2094658fead50dedd9c22c95c556a

On 23.12.18 13:59, Patrick Valsecchi wrote:


Hi,

Since haproxy version 1.8.14 and including the last 1.9 release,
haproxy puts all my backends in MAINT after around 31s. They
first work fine, but then they are put in MAINT.

The logs look like that:

<149>Dec 23 12:45:11 haproxy[1]: Proxy www started.
<149>Dec 23 12:45:11 haproxy[1]: Proxy plain started.
[NOTICE] 356/124511 (1) : New worker #1 (8) forked
<150>Dec 23 12:45:13 haproxy[8]: 89.217.194.174:49752
 [23/Dec/2018:12:45:13.098]
plain www/linked 0/0/16/21/37 200 4197 - -  1/1/0/0/0 0/0
"GET / HTTP/1.1"
[WARNING] 356/124542 (8) : Server www/linked is going DOWN
for maintenance (DNS timeout status). 0 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in
queue.
<145>Dec 23 12:45:42 haproxy[8]: Server www/linked is going
DOWN for maintenance (DNS timeout status). 0 active and 0
backup servers left. 0 sessions active, 0 requeued, 0
remaining in queue.
[ALERT] 356/124542 (8) : backend 'www' has no server available!
<144>Dec 23 12:45:42 haproxy[8]: backend www has no server
available!

I run haproxy using docker:

docker run --name toto -ti --rm -v
/home/docker-compositions/web/proxy/conf.test:/etc/haproxy/:ro
-p 8080:80 haproxy:1.9 haproxy -f /etc/haproxy/

And my config is that:

global
    log stderr local2
    chroot  /tmp
    pidfile /run/haproxy.pid
    maxconn 4000
    max-spread-checks 500

    master-worker

    user    nobody
    group   nogroup

resolvers dns
  nameserver docker 127.0.0.11:53 
  hold valid 1s

defaults
    mode    http
    log global
    option  httplog
    option  dontlognull
    option http-server-close
    option forwardfor   except 127.0.0.0/8

    option  redispatch
    retries 3
    timeout http-request    10s
    timeout queue   1m
    timeout connect 10s
    timeout client  10m
    timeout server  10m
    timeout http-keep-alive 10s
    timeout check   10s
    maxconn 3000
    default-server init-addr last,libc,none

    errorfile 400 /usr/local/etc/haproxy/errors/400.http
    errorfile 403 /usr/local/etc/haproxy/errors/403.http
    errorfile 408 /usr/local/etc/haproxy/errors/408.http
    errorfile 500 /usr/local/etc/haproxy/errors/500.http
    errorfile 502 /usr/local/etc/haproxy/errors/502.http
    errorfile 503 /usr/local/etc/haproxy/errors/503.http
    errorfile 504 /usr/local/etc/haproxy/errors/504.http

backend www
    option httpchk GET / HTTP/1.0\r\nUser-Agent:\ healthcheck
    http-check expect status 200
    default-server inter 60s fall 3 rise 1
    server linked www.topin.travel:80
 check resolvers dns

frontend plain
    bind :80

    http-request set-header X-Forwarded-Proto   http
    http-request set-header X-Forwarded-Host   
%[req.hdr(host)]
    http-request set-header X-Forwarded-Port    %[dst_port]
    http-request set-header X-Forwarded-For %[src]
    http-request set-header X-Real-IP   %[src]

    compression algo gzip
    compression type text/css text/html text/javascript
application/javascript text/plain text/xml application/json

    # Forward to the main linked container by default
    default_backend www


Any idea what is happening? I'

Re: DNS resolution problem since 1.8.14

2018-12-23 Thread Jonathan Matthews 
Hey Patrick,

Have you looked at the fixes in 1.8.16? They sound kinda-sorta related to
your problem ...

J

On Sun, 23 Dec 2018 at 16:17, Patrick Valsecchi  wrote:

> I did a tcpdump. My config is modified to point to a local container (www)
> in a docker compose (I'm trying to simplify my setup). You can see the DNS
> answers correctly:
>
> 16:06:00.181533 IP (tos 0x0, ttl 64, id 63816, offset 0, flags [DF], proto
> UDP (17), length 68)
> 127.0.0.11.53 > localhost.40994: 63037 1/0/0 www. A 172.20.0.17 (40)
>
> Could it be related to that?
> https://github.com/haproxy/haproxy/commit/8d4e7dc880d2094658fead50dedd9c22c95c556a
> On 23.12.18 13:59, Patrick Valsecchi wrote:
>
> Hi,
>
> Since haproxy version 1.8.14 and including the last 1.9 release, haproxy
> puts all my backends in MAINT after around 31s. They first work fine, but
> then they are put in MAINT.
>
> The logs look like that:
>
> <149>Dec 23 12:45:11 haproxy[1]: Proxy www started.
> <149>Dec 23 12:45:11 haproxy[1]: Proxy plain started.
> [NOTICE] 356/124511 (1) : New worker #1 (8) forked
> <150>Dec 23 12:45:13 haproxy[8]: 89.217.194.174:49752
> [23/Dec/2018:12:45:13.098] plain www/linked 0/0/16/21/37 200 4197 - - 
> 1/1/0/0/0 0/0 "GET / HTTP/1.1"
> [WARNING] 356/124542 (8) : Server www/linked is going DOWN for maintenance
> (DNS timeout status). 0 active and 0 backup servers left. 0 sessions
> active, 0 requeued, 0 remaining in queue.
> <145>Dec 23 12:45:42 haproxy[8]: Server www/linked is going DOWN for
> maintenance (DNS timeout status). 0 active and 0 backup servers left. 0
> sessions active, 0 requeued, 0 remaining in queue.
> [ALERT] 356/124542 (8) : backend 'www' has no server available!
> <144>Dec 23 12:45:42 haproxy[8]: backend www has no server available!
>
> I run haproxy using docker:
>
> docker run --name toto -ti --rm -v
> /home/docker-compositions/web/proxy/conf.test:/etc/haproxy/:ro -p 8080:80
> haproxy:1.9 haproxy -f /etc/haproxy/
>
> And my config is that:
>
> global
> log stderr local2
> chroot  /tmp
> pidfile /run/haproxy.pid
> maxconn 4000
> max-spread-checks 500
>
> master-worker
>
> usernobody
> group   nogroup
>
> resolvers dns
>   nameserver docker 127.0.0.11:53
>   hold valid 1s
>
> defaults
> modehttp
> log global
> option  httplog
> option  dontlognull
> option http-server-close
> option forwardfor   except 127.0.0.0/8
> option  redispatch
> retries 3
> timeout http-request10s
> timeout queue   1m
> timeout connect 10s
> timeout client  10m
> timeout server  10m
> timeout http-keep-alive 10s
> timeout check   10s
> maxconn 3000
> default-server init-addr last,libc,none
>
> errorfile 400 /usr/local/etc/haproxy/errors/400.http
> errorfile 403 /usr/local/etc/haproxy/errors/403.http
> errorfile 408 /usr/local/etc/haproxy/errors/408.http
> errorfile 500 /usr/local/etc/haproxy/errors/500.http
> errorfile 502 /usr/local/etc/haproxy/errors/502.http
> errorfile 503 /usr/local/etc/haproxy/errors/503.http
> errorfile 504 /usr/local/etc/haproxy/errors/504.http
>
> backend www
> option httpchk GET / HTTP/1.0\r\nUser-Agent:\ healthcheck
> http-check expect status 200
> default-server inter 60s fall 3 rise 1
> server linked www.topin.travel:80 check resolvers dns
>
> frontend plain
> bind :80
>
> http-request set-header X-Forwarded-Proto   http
> http-request set-header X-Forwarded-Host%[req.hdr(host)]
> http-request set-header X-Forwarded-Port%[dst_port]
> http-request set-header X-Forwarded-For %[src]
> http-request set-header X-Real-IP   %[src]
>
> compression algo gzip
> compression type text/css text/html text/javascript
> application/javascript text/plain text/xml application/json
>
> # Forward to the main linked container by default
> default_backend www
>
>
> Any idea what is happening? I've tried to increase the DNS resolve timeout
> to 5s and it didn't help. My feeling is that the newer versions of haproxy
> cannot talk with the DNS provided by docker.
>
> Thanks
>
> --
Jonathan Matthews
London, UK
http://www.jpluscplusm.com/contact.html

Re: DNS resolution problem since 1.8.14

2018-12-23 Thread Patrick Valsecchi 
I did a tcpdump. My config is modified to point to a local container 
(www) in a docker compose (I'm trying to simplify my setup). You can see 
the DNS answers correctly:


   16:06:00.181533 IP (tos 0x0, ttl 64, id 63816, offset 0, flags [DF],
   proto UDP (17), length 68)
    127.0.0.11.53 > localhost.40994: 63037 1/0/0 www. A 172.20.0.17
   (40)

Could it be related to that? 
https://github.com/haproxy/haproxy/commit/8d4e7dc880d2094658fead50dedd9c22c95c556a


On 23.12.18 13:59, Patrick Valsecchi wrote:


Hi,

Since haproxy version 1.8.14 and including the last 1.9 release, 
haproxy puts all my backends in MAINT after around 31s. They first 
work fine, but then they are put in MAINT.


The logs look like that:

<149>Dec 23 12:45:11 haproxy[1]: Proxy www started.
<149>Dec 23 12:45:11 haproxy[1]: Proxy plain started.
[NOTICE] 356/124511 (1) : New worker #1 (8) forked
<150>Dec 23 12:45:13 haproxy[8]: 89.217.194.174:49752
[23/Dec/2018:12:45:13.098] plain www/linked 0/0/16/21/37 200 4197
- -  1/1/0/0/0 0/0 "GET / HTTP/1.1"
[WARNING] 356/124542 (8) : Server www/linked is going DOWN for
maintenance (DNS timeout status). 0 active and 0 backup servers
left. 0 sessions active, 0 requeued, 0 remaining in queue.
<145>Dec 23 12:45:42 haproxy[8]: Server www/linked is going DOWN
for maintenance (DNS timeout status). 0 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 356/124542 (8) : backend 'www' has no server available!
<144>Dec 23 12:45:42 haproxy[8]: backend www has no server available!

I run haproxy using docker:

docker run --name toto -ti --rm -v
/home/docker-compositions/web/proxy/conf.test:/etc/haproxy/:ro -p
8080:80 haproxy:1.9 haproxy -f /etc/haproxy/

And my config is that:

global
    log stderr local2
    chroot  /tmp
    pidfile /run/haproxy.pid
    maxconn 4000
    max-spread-checks 500

    master-worker

    user    nobody
    group   nogroup

resolvers dns
  nameserver docker 127.0.0.11:53
  hold valid 1s

defaults
    mode    http
    log global
    option  httplog
    option  dontlognull
    option http-server-close
    option forwardfor   except 127.0.0.0/8
    option  redispatch
    retries 3
    timeout http-request    10s
    timeout queue   1m
    timeout connect 10s
    timeout client  10m
    timeout server  10m
    timeout http-keep-alive 10s
    timeout check   10s
    maxconn 3000
    default-server init-addr last,libc,none

    errorfile 400 /usr/local/etc/haproxy/errors/400.http
    errorfile 403 /usr/local/etc/haproxy/errors/403.http
    errorfile 408 /usr/local/etc/haproxy/errors/408.http
    errorfile 500 /usr/local/etc/haproxy/errors/500.http
    errorfile 502 /usr/local/etc/haproxy/errors/502.http
    errorfile 503 /usr/local/etc/haproxy/errors/503.http
    errorfile 504 /usr/local/etc/haproxy/errors/504.http

backend www
    option httpchk GET / HTTP/1.0\r\nUser-Agent:\ healthcheck
    http-check expect status 200
    default-server inter 60s fall 3 rise 1
    server linked www.topin.travel:80 check resolvers dns

frontend plain
    bind :80

    http-request set-header X-Forwarded-Proto http
    http-request set-header X-Forwarded-Host %[req.hdr(host)]
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request set-header X-Forwarded-For %[src]
    http-request set-header X-Real-IP %[src]

    compression algo gzip
    compression type text/css text/html text/javascript
application/javascript text/plain text/xml application/json

    # Forward to the main linked container by default
    default_backend www


Any idea what is happening? I've tried to increase the DNS resolve 
timeout to 5s and it didn't help. My feeling is that the newer 
versions of haproxy cannot talk with the DNS provided by docker.


Thanks