Re: 1.8.1 backend stays 'DOWN' when dns resolvers and http health checks are used

2017-12-19 Thread Holger Amann 

> Am 18.12.2017 um 15:52 schrieb Christopher Faulet :
> 
> There have been some fixes since the 1.8.1. One of them could fix your 
> problem: http://git.haproxy.org/?p=haproxy-1.8.git;a=commit;h=80b92902 
> 
Thanks Christopher, that fixes it!

Re: 1.8.1 backend stays 'DOWN' when dns resolvers and http health checks are used

2017-12-18 Thread Christopher Faulet 

Le 16/12/2017 à 00:22, Holger Amann a écrit :

Hi,

when I use dns resolvers combined with http health checks, the backend either 
never goes to 'UP' when haproxy starts before the backend, or the backend stays 
'DOWN' when it wasn't resolvable for a while.


--> starting haproxy:

[WARNING] 348/224658 (10) : Server phoenix-backend/phoenix is DOWN, reason: 
Layer4 timeout, check duration: 1001ms. 0 active and 0 backup servers left. 0 
sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 348/224658 (10) : backend 'phoenix-backend' has no server available!
[WARNING] 348/224727 (10) : Server phoenix-backend/phoenix was DOWN and now 
enters maintenance (DNS timeout status).

--> starting container:

[WARNING] 348/224914 (10) : phoenix-backend/phoenix changed its IP from 
46.38.243.234 to 172.19.0.7 by docker/docker.
[WARNING] 348/224914 (10) : Server phoenix-backend/phoenix administratively 
READY thanks to valid DNS answer.

then the process starts and is healthy, but backend stays 'down', no health 
checks got logged and I needed to restart haproxy which fixed everything.

--> stopping the process:

[WARNING] 348/225537 (10) : Server phoenix-backend/phoenix is DOWN, reason: Layer4 
connection problem, info: "Connection refused", check duration: 0ms. 0 active 
and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 348/225537 (10) : backend 'phoenix-backend' has no server available!

--> starting process:

[WARNING] 348/225735 (10) : Server phoenix-backend/phoenix is UP, reason: Layer7 check 
passed, code: 200, info: "OK", check duration: 171ms. 1 active and 0 backup 
servers online. 0 sessions requeued, 0 total in queue.

--> stopping process and container and waiting a few seconds until 'DOWN' 
message appears

[WARNING] 348/225847 (10) : Server phoenix-backend/phoenix is DOWN, reason: Layer4 
connection problem, info: "Connection refused", check duration: 0ms. 0 active 
and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 348/225847 (10) : backend 'phoenix-backend' has no serverver available!
[WARNING] 348/225923 (10) : Server phoenix-backend/phoenix was DOWN and now 
enters maintenance (DNS timeout status).

--> starting container again:
[WARNING] 348/230028 (10) : Server phoenix-backend/phoenix administratively 
READY thanks to valid DNS answer.

and starting the process, but nothing happens. again backend stays 'DOWN' and 
no health checks got logged.


haproxy -vv

HA-Proxy version 1.8.1 2017/12/03
Copyright 2000-2017 Willy Tarreau 

Build options :
   TARGET  = linux2628
   CPU = generic
   CC  = gcc
   CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv 
-Wno-null-dereference -Wno-unused-label
   OPTIONS = USE_LINUX_SPLICE=1 USE_LIBCRYPT=1 USE_ZLIB=1 USE_OPENSSL=1 
USE_PCRE=1

Default settings :
   maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.1.0f  25 May 2017
Running on OpenSSL version : OpenSSL 1.1.0f  25 May 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT 
IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), 
raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.

Available polling systems :
   epoll : pref=300,  test result OK
poll : pref=200,  test result OK
  select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
 [SPOE] spoe
 [COMP] compression
 [TRACE] trace



global
   log /dev/log local0
   log /dev/log local1 info

defaults
   log global
   mode http
   timeout connect 3s
   timeout client 30s
   timeout server 120s
   timeout tunnel 3600s
   timeout http-keep-alive  1s
   timeout http-request 15s
   option http-server-close
   option httplog
   option forwardfor
   errorfile 503 /code/docker/haproxy/503.html
   errorfile 408 /dev/null

resolvers docker
   nameserver docker 127.0.0.11:53
   hold valid 2s

listen stats
   bind :8444
   mode http
   stats enable
   stats realm Haproxy\ Statistics
   stats uri /

frontend http
   bind 0.0.0.0:80
   reqadd X-Forwarded-Proto:\ http

   default_backend phoenix-backend

frontend https
   bind 0.0.0.0:443 ssl crt "/code/certs/dev/dev-server-combined.pem" alpn 
h2,http/1.1 no-sslv3
   reqadd X-Forwarded-Proto:\ https

   default_backend phoenix-backend

backend phoenix-backend
   option httpchk GET /_check
   server phoenix phoenix:4000 resolvers docker init-addr libc,last,none check

1.8.1 backend stays 'DOWN' when dns resolvers and http health checks are used

2017-12-15 Thread Holger Amann 
Hi,

when I use dns resolvers combined with http health checks, the backend either 
never goes to 'UP' when haproxy starts before the backend, or the backend stays 
'DOWN' when it wasn't resolvable for a while.


--> starting haproxy:

[WARNING] 348/224658 (10) : Server phoenix-backend/phoenix is DOWN, reason: 
Layer4 timeout, check duration: 1001ms. 0 active and 0 backup servers left. 0 
sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 348/224658 (10) : backend 'phoenix-backend' has no server available!
[WARNING] 348/224727 (10) : Server phoenix-backend/phoenix was DOWN and now 
enters maintenance (DNS timeout status).

--> starting container:

[WARNING] 348/224914 (10) : phoenix-backend/phoenix changed its IP from 
46.38.243.234 to 172.19.0.7 by docker/docker.
[WARNING] 348/224914 (10) : Server phoenix-backend/phoenix administratively 
READY thanks to valid DNS answer.

then the process starts and is healthy, but backend stays 'down', no health 
checks got logged and I needed to restart haproxy which fixed everything.

--> stopping the process:

[WARNING] 348/225537 (10) : Server phoenix-backend/phoenix is DOWN, reason: 
Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 
active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in 
queue.
[ALERT] 348/225537 (10) : backend 'phoenix-backend' has no server available!

--> starting process:

[WARNING] 348/225735 (10) : Server phoenix-backend/phoenix is UP, reason: 
Layer7 check passed, code: 200, info: "OK", check duration: 171ms. 1 active and 
0 backup servers online. 0 sessions requeued, 0 total in queue.

--> stopping process and container and waiting a few seconds until 'DOWN' 
message appears

[WARNING] 348/225847 (10) : Server phoenix-backend/phoenix is DOWN, reason: 
Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 
active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in 
queue.
[ALERT] 348/225847 (10) : backend 'phoenix-backend' has no serverver available!
[WARNING] 348/225923 (10) : Server phoenix-backend/phoenix was DOWN and now 
enters maintenance (DNS timeout status).

--> starting container again:
[WARNING] 348/230028 (10) : Server phoenix-backend/phoenix administratively 
READY thanks to valid DNS answer.

and starting the process, but nothing happens. again backend stays 'DOWN' and 
no health checks got logged.

> haproxy -vv
HA-Proxy version 1.8.1 2017/12/03
Copyright 2000-2017 Willy Tarreau 

Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv 
-Wno-null-dereference -Wno-unused-label
  OPTIONS = USE_LINUX_SPLICE=1 USE_LIBCRYPT=1 USE_ZLIB=1 USE_OPENSSL=1 
USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.1.0f  25 May 2017
Running on OpenSSL version : OpenSSL 1.1.0f  25 May 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT 
IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), 
raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace



global
  log /dev/log local0
  log /dev/log local1 info

defaults
  log global
  mode http
  timeout connect 3s
  timeout client 30s
  timeout server 120s
  timeout tunnel 3600s
  timeout http-keep-alive  1s
  timeout http-request 15s
  option http-server-close
  option httplog
  option forwardfor
  errorfile 503 /code/docker/haproxy/503.html
  errorfile 408 /dev/null

resolvers docker
  nameserver docker 127.0.0.11:53
  hold valid 2s

listen stats
  bind :8444
  mode http
  stats enable
  stats realm Haproxy\ Statistics
  stats uri /

frontend http
  bind 0.0.0.0:80
  reqadd X-Forwarded-Proto:\ http

  default_backend phoenix-backend

frontend https
  bind 0.0.0.0:443 ssl crt "/code/certs/dev/dev-server-combined.pem" alpn 
h2,http/1.1 no-sslv3
  reqadd X-Forwarded-Proto:\ https

  default_backend phoenix-backend

backend phoenix-backend
  option httpchk GET /_check
  server phoenix phoenix:4000 resolvers docker init-addr libc,last,none check 
inter 1000