Re: About the 'Hot Restarts' of haproxy
On Wed, Apr 14, 2021 at 02:03:52AM +, Rmrf99 wrote: > Thanks Chris, Willy! this make me clear now. > Glad to see dynamically add/remove servers feature under development. If you're interested by with feature, please try the latest 2.4-dev, you already have the "add server" feature on the command line, you just need to enable "experimental-mode on" before accessing the command. Willy
Re: About the 'Hot Restarts' of haproxy
Thanks Chris, Willy! this make me clear now. Glad to see dynamically add/remove servers feature under development. ‐‐‐ Original Message ‐‐‐ On Wednesday, April 14, 2021 1:19 AM, Christopher Faulet wrote: > Le 13/04/2021 à 18:15, John Lauro a écrit : > > > Sounds like the biggest part of hot restarts is the cost of leaving the old > > process running as they have a lot of long running TCP connections, and if > > you > > do a lot of restarts the memory requirements build up. Not much of an > > issue for > > short lived http requests (although it would be nice if keep alive wasn't > > followed on the old haproxy processes so they could die quicker). > > Well, AFAIK, Envoy handles hot restarts exactly the same way HAProxy does. The > old process tries to finish to process in-flight requests. The connections are > not kept-alive. The old process closes all idle connections. But it must still > wait the responses for already started requests. Both Envoy and HAProxy do it > this way. Note there is an option to kill the old process after a given time. > > The article is a bit biased and inaccurate because it suggests HAproxy does > not > support hot restarts while Envoy do it. In fact, The real difference here is > the > ability to dynamically add or remove servers with Envoy. Thanks to this > feature, > most of time, there is no reason to restart it. Thus, hot restarts are not an > issue anymore. On HAProxy side, as Willy said, this feature is under > development. > > -- > > Christopher Faulet
Re: About the 'Hot Restarts' of haproxy
Le 13/04/2021 à 18:15, John Lauro a écrit : Sounds like the biggest part of hot restarts is the cost of leaving the old process running as they have a lot of long running TCP connections, and if you do a lot of restarts the memory requirements build up. Not much of an issue for short lived http requests (although it would be nice if keep alive wasn't followed on the old haproxy processes so they could die quicker). Well, AFAIK, Envoy handles hot restarts exactly the same way HAProxy does. The old process tries to finish to process in-flight requests. The connections are not kept-alive. The old process closes all idle connections. But it must still wait the responses for already started requests. Both Envoy and HAProxy do it this way. Note there is an option to kill the old process after a given time. The article is a bit biased and inaccurate because it suggests HAproxy does not support hot restarts while Envoy do it. In fact, The real difference here is the ability to dynamically add or remove servers with Envoy. Thanks to this feature, most of time, there is no reason to restart it. Thus, hot restarts are not an issue anymore. On HAProxy side, as Willy said, this feature is under development. -- Christopher Faulet
Re: About the 'Hot Restarts' of haproxy
On Tue, Apr 13, 2021 at 12:15:33PM -0400, John Lauro wrote: > Sounds like the biggest part of hot restarts is the cost of leaving the old > process running as they have a lot of long running TCP connections, and if > you do a lot of restarts the memory requirements build up. Not much of an > issue for short lived http requests (although it would be nice if keep > alive wasn't followed on the old haproxy processes so they could die > quicker). Exactly, which is why we've been working towards adding unlimited servers as this is essentially the only reason for restarting nowadays, e.g. when you need to add more servers than you had initially planned and already maxed out your server-template reserve :-) Willy
Re: About the 'Hot Restarts' of haproxy
Sounds like the biggest part of hot restarts is the cost of leaving the old process running as they have a lot of long running TCP connections, and if you do a lot of restarts the memory requirements build up. Not much of an issue for short lived http requests (although it would be nice if keep alive wasn't followed on the old haproxy processes so they could die quicker). On Tue, Apr 13, 2021 at 6:25 AM Willy Tarreau wrote: > On Tue, Apr 13, 2021 at 01:31:12AM +, Rmrf99 wrote: > > In this Slack engineering blog post: > https://slack.engineering/migrating-millions-of-concurrent-websockets-to-envoy/ > > > > they replace HAProxy with Envoy for **Hot Restart**, just curious does > > HAProxy new version will have similar approach? or have better > solution(in > > the future). > > So in fact it's not for hot restarts, since we've supported that even > before envoy was born, it's in order to introduce new servers at run > time. We do have some ongoing work on this, and some significant parts > are already available with experimental support: > > https://github.com/haproxy/haproxy/issues/1136 > > Willy > >
Re: About the 'Hot Restarts' of haproxy
On Tue, Apr 13, 2021 at 01:31:12AM +, Rmrf99 wrote: > In this Slack engineering blog post: > https://slack.engineering/migrating-millions-of-concurrent-websockets-to-envoy/ > > they replace HAProxy with Envoy for **Hot Restart**, just curious does > HAProxy new version will have similar approach? or have better solution(in > the future). So in fact it's not for hot restarts, since we've supported that even before envoy was born, it's in order to introduce new servers at run time. We do have some ongoing work on this, and some significant parts are already available with experimental support: https://github.com/haproxy/haproxy/issues/1136 Willy
