Re: Is it possible to avoid 503 error when one backend server has down and health check hasn't been launched yet

2016-12-25 Thread Alex.Chen
Thank you, Partrick and Willy,  your suggestions are very useful!

2016-12-25 1:46 GMT+08:00 Willy Tarreau :

> On Sat, Dec 24, 2016 at 05:16:53PM +, Patrick Hemmer wrote:
> >
> > On 2016/12/24 10:42, Alex.Chen wrote:
> > > for my scenario, i need to using "balance source" to keep the
> > > persistence of haproxy's balancing, I find that when one of my backend
> > > server (s1) has been killed, and if the next round health check is
> > > still not launched, then s1 is still be marked as UP. after 3 retries,
> > > the redispatch option does not work, I still get a 503 error. after a
> > > while, health check launched and s1 has been marked as DOWN, then my
> > > req has been forward to another backend server and everything is ok
> now.
> > >
> > > my quesition is that, is there any config can help me to avoid 503
> > > error when 3 retries have been failed but s1 is still marked as UP
> > > before the next round health check
> > >
> > > I debug  haproxy(1.6.10) and find that when I using "balance source",
> > >  the redispatch option does not work actually. after 3 retries,
> > > redispatch does not work, I guess that is because "balance source" is
> > > deterministic based on source IP and server state info(UP/DOWN and
> > > weight) (from
> > > : http://blog.haproxy.com/2013/04/22/client-ip-persistence-
> or-source-ip-hash-load-balancing/
> > > ) so if the server looks like "UP" then the balance source will still
> > > assign redispatch new conn to this deaded server s1.
> > >
> > >
> > I would think the "observe" option should handle this issue.
> > https://cbonte.github.io/haproxy-dconv/1.7/
> configuration.html#5.2-observe
>
> Yep definitely, it's one of its use cases. As a complement it is also
> recommended to set "fastinter" to a value much lower than "inter" so
> that once a first check fails, next health checks are triggered very
> quickly and the server is evicted fast.
>
> Willy
>


[ANNOUNCE] haproxy-1.5.19

2016-12-25 Thread Willy Tarreau
Hi,

HAProxy 1.5.19 was released on 2016/12/25. It added 47 new commits
after version 1.5.18.

[ before I forget, I'm running low on battery so I'll update the web site later 
]

This version fixes a number of severe issues affecting 1.5. On of them is
causing certain connections to become frozen forever if another connection
experienced a redispatch using the same file descriptor during a certain
time frame.

Another one appears when building with gcc 6, the listening IP address may
be ignored, resulting in the process listening to all addresses instead of
a single one.

Another bug may cause a runtime crash, when using sc_trackers with a
wrong table, a NULL pointer can be dereferenced.

We got a few reports of crashes in zlib not happening with slz, and the
bug was (as we guessed) indeed in haproxy, some unused fields had to be
initialized during the flush though it was not clearly documented.

And we (hopefully) fixed all the remaining systemd-related issues of
zombie processes and incorrect return codes.

The remaining ones are less important (or at least avoidable in normal
conditions).

I know it's been a long time without a release (7 months), so if you're
running on a snapshot between 1.5.18 and 1.5.19, you may additionally be
exposed to some temporary regressions that happened while trying to fix
the redispatch issue above that have since been fixed, thus it's important
to upgrade.

Please find the usual URLs below :
   Site index   : http://www.haproxy.org/
   Discourse: http://discourse.haproxy.org/
   Sources  : http://www.haproxy.org/download/1.5/src/
   Git repository   : http://git.haproxy.org/git/haproxy-1.5.git/
   Git Web browsing : http://git.haproxy.org/?p=haproxy-1.5.git
   Changelog: http://www.haproxy.org/download/1.5/src/CHANGELOG
   Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/

Willy
---
Complete changelog :
  - BUG/MAJOR: fix listening IP address storage for frontends
  - CLEANUP: connection: fix double negation on memcmp()
  - BUG/MEDIUM: sticktables: segfault in some configuration error cases
  - BUG/MINOR: http: add-header: header name copied twice
  - BUG/MINOR: ssl: fix potential memory leak in ssl_sock_load_dh_params()
  - BUG/MINOR: http: url32+src should use the big endian version of url32
  - BUG/MINOR: http: url32+src should check cli_conn before using it
  - DOC: http: add documentation for url32 and url32+src
  - MINOR: systemd: Use variable for config and pidfile paths
  - MINOR: systemd: Perform sanity check on config before reload
  - BUG/MINOR: init: always ensure that global.rlimit_nofile matches actual 
limits
  - BUG/MINOR: init: ensure that FD limit is raised to the max allowed
  - Revert "BUG/MINOR: ssl: fix potential memory leak in 
ssl_sock_load_dh_params()"
  - BUG/MEDIUM: stream-int: completely detach connection on connect error
  - DOC: minor typo fixes to improve HTML parsing by haproxy-dconv
  - BUG/MAJOR: compression: initialize avail_in/next_in even during flush
  - BUG/MAJOR: stick-counters: possible crash when using sc_trackers with wrong 
table
  - BUG/MAJOR: stream: properly mark the server address as unset on connect 
retry
  - BUG/MINOR: payload: fix SSLv2 version parser
  - MINOR: cli: allow the semi-colon to be escaped on the CLI
  - BUG/MINOR: displayed PCRE version is running release
  - MINOR: show Built with PCRE version
  - MINOR: show Running on zlib version
  - BUG/MINOR: ssl: Check malloc return code
  - BUG/MINOR: ssl: prevent multiple entries for the same certificate
  - BUG/MINOR: systemd: make the wrapper return a non-null status code on error
  - BUILD/CLEANUP: systemd: avoid a warning due to mixed code and declaration
  - BUG/MINOR: systemd: always restore signals before execve()
  - BUG/MINOR: systemd: check return value of calloc()
  - MINOR: systemd: report it when execve() fails
  - BUG/MEDIUM: systemd: let the wrapper know that haproxy has completed or 
failed
  - BUILD: poll: remove unused hap_fd_isset() which causes a warning with clang
  - DOC: Fix typo in description of `-st` parameter in man page
  - BUG/MEDIUM: peers: fix use after free in peer_session_create()
  - BUG/MEDIUM: systemd-wrapper: return correct exit codes
  - BUG/MINOR: stick-table: handle out-of-memory condition gracefully
  - BUG/MEDIUM: connection: check the control layer before stopping polling
  - BUG/MEDIUM: stick-table: fix regression caused by recent fix for 
out-of-memory
  - BUG/MINOR: cli: properly decrement ref count on tables during failed dumps
  - BUG/MINOR: cli: fix pointer size when reporting data/transport layer name
  - BUG/MINOR: cli: dequeue from the proxy when changing a maxconn
  - BUG/MEDIUM: proxy: return "none" and "unknown" for unknown LB algos
  - BUG/MINOR: http: don't send an extra CRLF after a Set-Cookie in a redirect
  - DOC: fix small typo in fe_id (backend instead of frontend)
  - BUG/MEDIUM: ssl: properly reset the reused_sess during a forced handshake
  - BUG/MINOR: backend: 

[ANNOUNCE] haproxy-1.6.11

2016-12-25 Thread Willy Tarreau
Hi,

HAProxy 1.6.11 was released on 2016/12/25. It added 24 new commits
after version 1.6.10.

The most important changes are fixes for out-of-memory conditions, possibly
causing some applets to remain stuck when waiting for a buffer to be available
(possibly impacting peers, Lua or CLI). Some inter-task signals could also be
lost in such a condition, possibly causing some delays with cosockets in Lua.
The server-side SNI appeared to be unreliable because once a connection was
established to the server, the same SSL session was reused as long as possible,
thus with the same SNI. Not only would it interact badly with helth checks, but
also with multiple SNIs to the server. The rest is the usual bag of small
harmless issues.

If 1.6.10 works fine for you and you're not running in memory-constrained
environments, there's no urge to upgrade. But it doesn't hurt anyway :-)

Please find the usual URLs below :
   Site index   : http://www.haproxy.org/
   Discourse: http://discourse.haproxy.org/
   Sources  : http://www.haproxy.org/download/1.6/src/
   Git repository   : http://git.haproxy.org/git/haproxy-1.6.git/
   Git Web browsing : http://git.haproxy.org/?p=haproxy-1.6.git
   Changelog: http://www.haproxy.org/download/1.6/src/CHANGELOG
   Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/

Willy
---
Complete changelog :
  - BUILD: contrib: fix ip6range build on Centos 7
  - BUG/MINOR: cli: fix pointer size when reporting data/transport layer name
  - BUG/MINOR: cli: dequeue from the proxy when changing a maxconn
  - BUG/MINOR: cli: wake up the CLI's task after a timeout update
  - BUG/MINOR: freq-ctr: make swrate_add() support larger values
  - BUG/MEDIUM: proxy: return "none" and "unknown" for unknown LB algos
  - BUG/MAJOR: stream: fix session abort on resource shortage
  - BUG/MINOR: http: don't send an extra CRLF after a Set-Cookie in a redirect
  - BUG/MEDIUM: variables: some variable name can hide another ones
  - BUG/MINOR: cli: be sure to always warn the cli applet when input buffer is 
full
  - MINOR: applet: Count number of (active) applets
  - MINOR: task: Rename run_queue and run_queue_cur counters
  - BUG/MEDIUM: stream: Save unprocessed events for a stream
  - BUG/MAJOR: Fix how the list of entities waiting for a buffer is handled
  - BUG/MEDIUM: lua: In some case, the return of sample-fetches is ignored (2)
  - BUG/MINOR: stream-int: automatically release SI_FL_WAIT_DATA on SHUTW_NOW
  - DOC: lua: section declared twice
  - DOC: fix small typo in fe_id (backend instead of frontend)
  - BUG/MINOR: lua: memory leak executing tasks
  - BUG/MEDIUM: ssl: properly reset the reused_sess during a forced handshake
  - BUG/MEDIUM: ssl: avoid double free when releasing bind_confs
  - BUG/MINOR: backend: nbsrv() should return 0 if backend is disabled
  - BUG/MEDIUM: ssl: for a handshake when server-side SNI changes
  - BUG/MINOR: systemd: potential zombie processes
---