Re: Cannot enable a config "disabled" server via socket command
On Fri, Sep 18, 2015 at 01:24:59PM +0200, Baptiste wrote: > I'm using it carefully. I set up a matrix of all combination from old > config, state at run time in old process and state in new config and > came to the conclusion that: > - in case of a configuration change, we apply the setting from the > new configuration, regardless of old running state (I'll check if the > documentation also relates this behavior) > - if no configuration change, we apply old running state only if old > running state is different from new configuration state Yes that's the principle. > Actually, the same apply to the server's iweight parameter to decide > about DRAIN state and computation of eweight. Indeed. > Looking forward to send you the patches regarding server states in a > few minutes. OK fine. I have a kernel to release right now and after that I can switch to your patches. Willy
Re: Cannot enable a config "disabled" server via socket command
On Fri, Sep 18, 2015 at 12:40 PM, Willy Tarreau wrote: > On Fri, Sep 18, 2015 at 10:51:13AM +0200, Baptiste wrote: >> Please find a patch in attachment which fix the issue so you can run >> temporarly the latest code in production. >> That said, I need to discuss with Willy to ensure this is the best way >> to do it, since this patch changes the design we did. >> Anyway, I've updated my server-state code to match the new behavior, >> soe everyone should be satisfied. > > I totally agree with your patch Baptiste and I've merged it. It > does what we initially designed, I was the one who got confused > by it during the first review (in part due to too much time > between design and review). Sorry for that. Keep in mind you'll > have to be careful in your reload patch to properly consider > FMAINT and CMAINT separately. > > Thanks, > Willy > Ok, thanks Willy. I'm using it carefully. I set up a matrix of all combination from old config, state at run time in old process and state in new config and came to the conclusion that: - in case of a configuration change, we apply the setting from the new configuration, regardless of old running state (I'll check if the documentation also relates this behavior) - if no configuration change, we apply old running state only if old running state is different from new configuration state Actually, the same apply to the server's iweight parameter to decide about DRAIN state and computation of eweight. Looking forward to send you the patches regarding server states in a few minutes. Baptiste
Re: Cannot enable a config "disabled" server via socket command
On Fri, Sep 18, 2015 at 10:51:13AM +0200, Baptiste wrote: > Please find a patch in attachment which fix the issue so you can run > temporarly the latest code in production. > That said, I need to discuss with Willy to ensure this is the best way > to do it, since this patch changes the design we did. > Anyway, I've updated my server-state code to match the new behavior, > soe everyone should be satisfied. I totally agree with your patch Baptiste and I've merged it. It does what we initially designed, I was the one who got confused by it during the first review (in part due to too much time between design and review). Sorry for that. Keep in mind you'll have to be careful in your reload patch to properly consider FMAINT and CMAINT separately. Thanks, Willy
Re: Cannot enable a config "disabled" server via socket command
Cool, thanks Baptiste. Appreciate it. We will check the patch out. On Sep 18, 2015 2:21 PM, "Baptiste" wrote: > Please find a patch in attachment which fix the issue so you can run > temporarly the latest code in production. > That said, I need to discuss with Willy to ensure this is the best way > to do it, since this patch changes the design we did. > Anyway, I've updated my server-state code to match the new behavior, > soe everyone should be satisfied. > > Baptiste > > On Fri, Sep 18, 2015 at 5:28 AM, Pradeep Jindal > wrote: > > It'd be interesting to know the complete semantics of the feature you are > > implementing. I know you understand that our use case is a valid one. > And we > > are open to explore alternative approaches to achieve the same. Till > then we > > will be running haproxy with the flag change reverted and wait for your > > response on this. > > > > On Sep 18, 2015 1:25 AM, "Baptiste" wrote: > >> > >> On Thu, Sep 17, 2015 at 9:42 PM, Pavlos Parissis > >> wrote: > >> > On 15/09/2015 08:45 πμ, Cyril Bonté wrote: > >> >> Hi, > >> >> > >> >> > >> >> Le 14/09/2015 14:23, Ayush Goyal a écrit : > >> >>> Hi, > >> >>> > >> >>> We are testing haproxy-1.6dev4, we have added a server in backend as > >> >>> disabled, but we are not able > >> >>> to bring it up using socket command. > >> >>> > >> >>> Our backend conf looks like this: > >> >>> > >> >>> =cut > >> >>> backend apiservers > >> >>> server api101 localhost:1234 maxconn 128 weight 1 > >> >>> check > >> >>> server api102 localhost:1235 disabled maxconn 128 weight 1 > >> >>> check > >> >>> server api103 localhost:1236 disabled maxconn 128 weight 1 > >> >>> check > >> >>> =cut > >> >>> > >> >>> But, when I run the "enable apiservers/api103" command, it is still > in > >> >>> MAINT mode. Disabling and enabling of non "disabled" servers like > >> >>> api101 > >> >>> are happening properly. > >> >>> > >> >>> Enabling a config "disabled" server works correctly with haproxy1.5. > >> >>> Can > >> >>> you confirm whether its a bug in 1.6-dev4? > >> >> > >> >> This is due to the introduction of the SRV_ADMF_CMAINT flag, which is > >> >> set permanently. The "enable/disable" socket command will only modify > >> >> the SRV_ADMF_FMAINT and SRV_ADMF_FDRAIN flags. > >> >> > >> >> I add Baptiste to the thread. > >> >> > >> > > >> > That will break our setup as well, where an external tool uses the > >> > socket to disable a server in the running config and regenerate the > >> > configuration with the server disabled. > >> > > >> > I am also interested in knowing the motivation behind this change. > >> > > >> > Cheers, > >> > Pavlos > >> > > >> > > >> > >> Hi all, > >> > >> This "feature" was an early patch for later coming feature to avoid > >> any impact of reloading HAProxy on servers state. > >> I'm currently finishing the dev before forwarding the patches to Willy > >> by tomorrow. > >> We needed to know the real reason why a server was in maintenance > >> state: was it because of configuration or through the socket, so at > >> next reload we could apply the right state based on old running state, > >> old config state and new config state. > >> > >> I'm going to check what we can do to fix your issue. > >> > >> Baptiste > >> > > >
Re: Cannot enable a config "disabled" server via socket command
Please find a patch in attachment which fix the issue so you can run
temporarly the latest code in production.
That said, I need to discuss with Willy to ensure this is the best way
to do it, since this patch changes the design we did.
Anyway, I've updated my server-state code to match the new behavior,
soe everyone should be satisfied.
Baptiste
On Fri, Sep 18, 2015 at 5:28 AM, Pradeep Jindal wrote:
> It'd be interesting to know the complete semantics of the feature you are
> implementing. I know you understand that our use case is a valid one. And we
> are open to explore alternative approaches to achieve the same. Till then we
> will be running haproxy with the flag change reverted and wait for your
> response on this.
>
> On Sep 18, 2015 1:25 AM, "Baptiste" wrote:
>>
>> On Thu, Sep 17, 2015 at 9:42 PM, Pavlos Parissis
>> wrote:
>> > On 15/09/2015 08:45 πμ, Cyril Bonté wrote:
>> >> Hi,
>> >>
>> >>
>> >> Le 14/09/2015 14:23, Ayush Goyal a écrit :
>> >>> Hi,
>> >>>
>> >>> We are testing haproxy-1.6dev4, we have added a server in backend as
>> >>> disabled, but we are not able
>> >>> to bring it up using socket command.
>> >>>
>> >>> Our backend conf looks like this:
>> >>>
>> >>> =cut
>> >>> backend apiservers
>> >>> server api101 localhost:1234 maxconn 128 weight 1
>> >>> check
>> >>> server api102 localhost:1235 disabled maxconn 128 weight 1
>> >>> check
>> >>> server api103 localhost:1236 disabled maxconn 128 weight 1
>> >>> check
>> >>> =cut
>> >>>
>> >>> But, when I run the "enable apiservers/api103" command, it is still in
>> >>> MAINT mode. Disabling and enabling of non "disabled" servers like
>> >>> api101
>> >>> are happening properly.
>> >>>
>> >>> Enabling a config "disabled" server works correctly with haproxy1.5.
>> >>> Can
>> >>> you confirm whether its a bug in 1.6-dev4?
>> >>
>> >> This is due to the introduction of the SRV_ADMF_CMAINT flag, which is
>> >> set permanently. The "enable/disable" socket command will only modify
>> >> the SRV_ADMF_FMAINT and SRV_ADMF_FDRAIN flags.
>> >>
>> >> I add Baptiste to the thread.
>> >>
>> >
>> > That will break our setup as well, where an external tool uses the
>> > socket to disable a server in the running config and regenerate the
>> > configuration with the server disabled.
>> >
>> > I am also interested in knowing the motivation behind this change.
>> >
>> > Cheers,
>> > Pavlos
>> >
>> >
>>
>> Hi all,
>>
>> This "feature" was an early patch for later coming feature to avoid
>> any impact of reloading HAProxy on servers state.
>> I'm currently finishing the dev before forwarding the patches to Willy
>> by tomorrow.
>> We needed to know the real reason why a server was in maintenance
>> state: was it because of configuration or through the socket, so at
>> next reload we could apply the right state based on old running state,
>> old config state and new config state.
>>
>> I'm going to check what we can do to fix your issue.
>>
>> Baptiste
>>
>
From 752f5d2f02951a3c271c27a6e3a202ef7ada0b73 Mon Sep 17 00:00:00 2001
From: Baptiste Assmann
Date: Fri, 18 Sep 2015 10:30:03 +0200
Subject: [PATCH 01/12] BUG/MAJOR: can't enable a server through the stat
socket
When a server is disabled in the configuration using the "disabled"
keyword, a single flag is positionned: SRV_ADMF_CMAINT (use to be
SRV_ADMF_FMAINT)..
That said, when providing the first version of this code, we also
changed the SRV_ADMF_MAINT mask to match any of the possible MAINT
cases: SRV_ADMF_FMAINT, SRV_ADMF_IMAINT, SRV_ADMF_CMAINT
Since SRV_ADMF_CMAINT is never (and is not supposed to be) altered at
run time, once a server has this flag set up, it can never ever be
enabled again using the stats socket.
In order to fix this, we should:
- consider SRV_ADMF_CMAINT as a simple flag to report the state in the
old configuration file (will be used after a reload to deduce the
state of the server in a new running process)
- enabling both SRV_ADMF_CMAINT and SRV_ADMF_FMAINT when the keyword
"disabled" is in use in the configuration
- update the mask SRV_ADMF_MAINT as it was before, to only match
SRV_ADMF_FMAINT and SRV_ADMF_IMAINT.
The following patch perform the changes above.
It allows fixing the regression without breaking the way the up coming
feature (seamless server state accross reloads) is going to work.
---
include/types/server.h | 2 +-
src/server.c | 1 +
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/include/types/server.h b/include/types/server.h
index ee31794..40b8308 100644
--- a/include/types/server.h
+++ b/include/types/server.h
@@ -75,8 +75,8 @@ enum srv_state {
enum srv_admin {
SRV_ADMF_FMAINT= 0x01,/* the server was explicitly forced into maintenance */
SRV_ADMF_IMAINT= 0x02,/* the server has inherited the maintenance status from a tracked server */
+ SRV_ADMF_MAINT = 0x03,/* mask to check if any maintenance flag is present */
SRV_ADMF_CMAIN
Re: Cannot enable a config "disabled" server via socket command
It'd be interesting to know the complete semantics of the feature you are implementing. I know you understand that our use case is a valid one. And we are open to explore alternative approaches to achieve the same. Till then we will be running haproxy with the flag change reverted and wait for your response on this. On Sep 18, 2015 1:25 AM, "Baptiste" wrote: > On Thu, Sep 17, 2015 at 9:42 PM, Pavlos Parissis > wrote: > > On 15/09/2015 08:45 πμ, Cyril Bonté wrote: > >> Hi, > >> > >> > >> Le 14/09/2015 14:23, Ayush Goyal a écrit : > >>> Hi, > >>> > >>> We are testing haproxy-1.6dev4, we have added a server in backend as > >>> disabled, but we are not able > >>> to bring it up using socket command. > >>> > >>> Our backend conf looks like this: > >>> > >>> =cut > >>> backend apiservers > >>> server api101 localhost:1234 maxconn 128 weight 1 > >>> check > >>> server api102 localhost:1235 disabled maxconn 128 weight 1 > >>> check > >>> server api103 localhost:1236 disabled maxconn 128 weight 1 > >>> check > >>> =cut > >>> > >>> But, when I run the "enable apiservers/api103" command, it is still in > >>> MAINT mode. Disabling and enabling of non "disabled" servers like > api101 > >>> are happening properly. > >>> > >>> Enabling a config "disabled" server works correctly with haproxy1.5. > Can > >>> you confirm whether its a bug in 1.6-dev4? > >> > >> This is due to the introduction of the SRV_ADMF_CMAINT flag, which is > >> set permanently. The "enable/disable" socket command will only modify > >> the SRV_ADMF_FMAINT and SRV_ADMF_FDRAIN flags. > >> > >> I add Baptiste to the thread. > >> > > > > That will break our setup as well, where an external tool uses the > > socket to disable a server in the running config and regenerate the > > configuration with the server disabled. > > > > I am also interested in knowing the motivation behind this change. > > > > Cheers, > > Pavlos > > > > > > Hi all, > > This "feature" was an early patch for later coming feature to avoid > any impact of reloading HAProxy on servers state. > I'm currently finishing the dev before forwarding the patches to Willy > by tomorrow. > We needed to know the real reason why a server was in maintenance > state: was it because of configuration or through the socket, so at > next reload we could apply the right state based on old running state, > old config state and new config state. > > I'm going to check what we can do to fix your issue. > > Baptiste > >
Re: Cannot enable a config "disabled" server via socket command
On 17/09/2015 09:53 μμ, Baptiste wrote: > On Thu, Sep 17, 2015 at 9:42 PM, Pavlos Parissis > wrote: >> On 15/09/2015 08:45 πμ, Cyril Bonté wrote: >>> Hi, >>> >>> >>> Le 14/09/2015 14:23, Ayush Goyal a écrit : Hi, We are testing haproxy-1.6dev4, we have added a server in backend as disabled, but we are not able to bring it up using socket command. Our backend conf looks like this: =cut backend apiservers server api101 localhost:1234 maxconn 128 weight 1 check server api102 localhost:1235 disabled maxconn 128 weight 1 check server api103 localhost:1236 disabled maxconn 128 weight 1 check =cut But, when I run the "enable apiservers/api103" command, it is still in MAINT mode. Disabling and enabling of non "disabled" servers like api101 are happening properly. Enabling a config "disabled" server works correctly with haproxy1.5. Can you confirm whether its a bug in 1.6-dev4? >>> >>> This is due to the introduction of the SRV_ADMF_CMAINT flag, which is >>> set permanently. The "enable/disable" socket command will only modify >>> the SRV_ADMF_FMAINT and SRV_ADMF_FDRAIN flags. >>> >>> I add Baptiste to the thread. >>> >> >> That will break our setup as well, where an external tool uses the >> socket to disable a server in the running config and regenerate the >> configuration with the server disabled. >> >> I am also interested in knowing the motivation behind this change. >> >> Cheers, >> Pavlos >> >> > > Hi all, > > This "feature" was an early patch for later coming feature to avoid > any impact of reloading HAProxy on servers state. > I'm currently finishing the dev before forwarding the patches to Willy > by tomorrow. > We needed to know the real reason why a server was in maintenance > state: was it because of configuration or through the socket, so at > next reload we could apply the right state based on old running state, > old config state and new config state. > Excellent approach. Cheers, Pavlos signature.asc Description: OpenPGP digital signature
Re: Cannot enable a config "disabled" server via socket command
On Thu, Sep 17, 2015 at 9:42 PM, Pavlos Parissis wrote: > On 15/09/2015 08:45 πμ, Cyril Bonté wrote: >> Hi, >> >> >> Le 14/09/2015 14:23, Ayush Goyal a écrit : >>> Hi, >>> >>> We are testing haproxy-1.6dev4, we have added a server in backend as >>> disabled, but we are not able >>> to bring it up using socket command. >>> >>> Our backend conf looks like this: >>> >>> =cut >>> backend apiservers >>> server api101 localhost:1234 maxconn 128 weight 1 >>> check >>> server api102 localhost:1235 disabled maxconn 128 weight 1 >>> check >>> server api103 localhost:1236 disabled maxconn 128 weight 1 >>> check >>> =cut >>> >>> But, when I run the "enable apiservers/api103" command, it is still in >>> MAINT mode. Disabling and enabling of non "disabled" servers like api101 >>> are happening properly. >>> >>> Enabling a config "disabled" server works correctly with haproxy1.5. Can >>> you confirm whether its a bug in 1.6-dev4? >> >> This is due to the introduction of the SRV_ADMF_CMAINT flag, which is >> set permanently. The "enable/disable" socket command will only modify >> the SRV_ADMF_FMAINT and SRV_ADMF_FDRAIN flags. >> >> I add Baptiste to the thread. >> > > That will break our setup as well, where an external tool uses the > socket to disable a server in the running config and regenerate the > configuration with the server disabled. > > I am also interested in knowing the motivation behind this change. > > Cheers, > Pavlos > > Hi all, This "feature" was an early patch for later coming feature to avoid any impact of reloading HAProxy on servers state. I'm currently finishing the dev before forwarding the patches to Willy by tomorrow. We needed to know the real reason why a server was in maintenance state: was it because of configuration or through the socket, so at next reload we could apply the right state based on old running state, old config state and new config state. I'm going to check what we can do to fix your issue. Baptiste
Re: Cannot enable a config "disabled" server via socket command
On 15/09/2015 08:45 πμ, Cyril Bonté wrote: > Hi, > > > Le 14/09/2015 14:23, Ayush Goyal a écrit : >> Hi, >> >> We are testing haproxy-1.6dev4, we have added a server in backend as >> disabled, but we are not able >> to bring it up using socket command. >> >> Our backend conf looks like this: >> >> =cut >> backend apiservers >> server api101 localhost:1234 maxconn 128 weight 1 >> check >> server api102 localhost:1235 disabled maxconn 128 weight 1 >> check >> server api103 localhost:1236 disabled maxconn 128 weight 1 >> check >> =cut >> >> But, when I run the "enable apiservers/api103" command, it is still in >> MAINT mode. Disabling and enabling of non "disabled" servers like api101 >> are happening properly. >> >> Enabling a config "disabled" server works correctly with haproxy1.5. Can >> you confirm whether its a bug in 1.6-dev4? > > This is due to the introduction of the SRV_ADMF_CMAINT flag, which is > set permanently. The "enable/disable" socket command will only modify > the SRV_ADMF_FMAINT and SRV_ADMF_FDRAIN flags. > > I add Baptiste to the thread. > That will break our setup as well, where an external tool uses the socket to disable a server in the running config and regenerate the configuration with the server disabled. I am also interested in knowing the motivation behind this change. Cheers, Pavlos signature.asc Description: OpenPGP digital signature
Re: Cannot enable a config "disabled" server via socket command
Hi, Could you please tell us about the motivation behind introducing SRV_ADMF_CMAINT flag? We were planning to avoid haproxy reloads by adding "dummy" servers in the configuration in "disabled" mode and then "set server addr" and "enable" socket commands to enable them on-demand. And that, we now won't be able to do with the introduction of SRV_ADMF_CMAINT flag. We want to avoid reloads because we have persistent TCP connections being served via haproxy and with a reload the previous haproxy process keeps on serving the old connections and we lose stats for those connections as the new process is not aware of them for obvious reasons. Avoiding reload also avoids stats counters being reset. - Pradeep Jindal On Tue, Sep 15, 2015 at 12:15 PM, Cyril Bonté wrote: > Hi, > > > Le 14/09/2015 14:23, Ayush Goyal a écrit : > >> Hi, >> >> We are testing haproxy-1.6dev4, we have added a server in backend as >> disabled, but we are not able >> to bring it up using socket command. >> >> Our backend conf looks like this: >> >> =cut >> backend apiservers >> server api101 localhost:1234 maxconn 128 weight 1 check >> server api102 localhost:1235 disabled maxconn 128 weight 1 check >> server api103 localhost:1236 disabled maxconn 128 weight 1 check >> =cut >> >> But, when I run the "enable apiservers/api103" command, it is still in >> MAINT mode. Disabling and enabling of non "disabled" servers like api101 >> are happening properly. >> >> Enabling a config "disabled" server works correctly with haproxy1.5. Can >> you confirm whether its a bug in 1.6-dev4? >> > > This is due to the introduction of the SRV_ADMF_CMAINT flag, which is set > permanently. The "enable/disable" socket command will only modify the > SRV_ADMF_FMAINT and SRV_ADMF_FDRAIN flags. > > I add Baptiste to the thread. > > -- > Cyril Bonté > >
Re: Cannot enable a config "disabled" server via socket command
Hi, Le 14/09/2015 14:23, Ayush Goyal a écrit : Hi, We are testing haproxy-1.6dev4, we have added a server in backend as disabled, but we are not able to bring it up using socket command. Our backend conf looks like this: =cut backend apiservers server api101 localhost:1234 maxconn 128 weight 1 check server api102 localhost:1235 disabled maxconn 128 weight 1 check server api103 localhost:1236 disabled maxconn 128 weight 1 check =cut But, when I run the "enable apiservers/api103" command, it is still in MAINT mode. Disabling and enabling of non "disabled" servers like api101 are happening properly. Enabling a config "disabled" server works correctly with haproxy1.5. Can you confirm whether its a bug in 1.6-dev4? This is due to the introduction of the SRV_ADMF_CMAINT flag, which is set permanently. The "enable/disable" socket command will only modify the SRV_ADMF_FMAINT and SRV_ADMF_FDRAIN flags. I add Baptiste to the thread. -- Cyril Bonté
Cannot enable a config "disabled" server via socket command
Hi, We are testing haproxy-1.6dev4, we have added a server in backend as disabled, but we are not able to bring it up using socket command. Our backend conf looks like this: =cut backend apiservers server api101 localhost:1234 maxconn 128 weight 1 check server api102 localhost:1235 disabled maxconn 128 weight 1 check server api103 localhost:1236 disabled maxconn 128 weight 1 check =cut But, when I run the "enable apiservers/api103" command, it is still in MAINT mode. Disabling and enabling of non "disabled" servers like api101 are happening properly. Enabling a config "disabled" server works correctly with haproxy1.5. Can you confirm whether its a bug in 1.6-dev4? Thanks, Ayush Goyal
