Re: haproxy bug: healthcheck not passing after port change when statefile is enabled
Hi, After debriefing internally, the fix will be much longer and may even trigger a new server-state file format. I keep you updated. Baptiste On Sun, Nov 4, 2018 at 7:11 PM Baptiste wrote: > Hi Sven, > > I reviewed the whole thing and I think the support of port in state file > was added for SRV records, but also for the runtime api, which allows > changing the port at runtime too. > I'll come back to you shortly with a fix for this behavior, currently > discussing with Willy/Fred about it. > (it's more complicated than moving the code > """ > if (port_str) > srv->svc_port = port; > """ > a couple of lines above). > > Baptiste > > > On Tue, Oct 9, 2018 at 10:52 AM Sven Wiltink wrote: > >> Hey Baptiste, >> >> >> We noticed the SRV patch has been merged. That should mean that we can >> now fix this issue as well. Would you be able to fix this or should we >> >> try to provide a patch? >> >> >> Thanks again in advance, >> >> Sven >> ------------------ >> *Van:* Baptiste >> *Verzonden:* donderdag 12 juli 2018 14:52:24 >> *Aan:* Sven Wiltink >> *CC:* haproxy@formilux.org >> *Onderwerp:* Re: haproxy bug: healthcheck not passing after port change >> when statefile is enabled >> >> Hi Sven, >> >> Thanks for the clarification. >> It's a bit more complicated than what it is supposed to be. >> I think we may want to apply the port only if it has been changed at >> runtime (changed by DNS SRV records). >> >> The status is the following: I have a pending patch which brings SRV >> record information into the state file. (WIP, but last mile) >> Once it has been merged, we'll be able to fix this issue (by applying the >> port only when the server is being managed by an SRV record). >> >> Baptiste >> >> >> On Tue, Jul 3, 2018 at 3:41 PM, Sven Wiltink wrote: >> >> Hey Baptiste, >> >> >> Thank you for looking into it. >> >> >> The bug is triggered by running haproxy with the following config: >> >> >> global >> maxconn 32000 >> tune.maxrewrite 2048 >> user haproxy >> group haproxy >> daemon >> chroot /var/lib/haproxy >> nbproc 1 >> maxcompcpuusage 85 >> spread-checks 0 >> stats socket /var/run/haproxy.sock mode 600 level admin process 1 >> user haproxy group haproxy >> server-state-file test >> server-state-base /var/run/haproxy/state >> master-worker no-exit-on-failure >> >> defaults >> load-server-state-from-file global >> log global >> timeout http-request 5s >> timeout connect 2s >> timeout client 300s >> timeout server 300s >> mode http >> option dontlog-normal >> option http-server-close >> option redispatch >> option log-health-checks >> >> listen stats >> bind :1936 >> bind-process 1 >> mode http >> stats enable >> stats uri / >> stats admin if TRUE >> >> listen banaan-443-ipv4 >> bind :443 >> mode tcp >> server banaan-vps 127.0.0.1:443 check inter 2000 >> >> >> - Then start haproxy (it will do healthchecks to port 443) >> - change server banaan-vps 127.0.0.1:443 check inter 2000 to server >> banaan-vps 127.0.0.1:80 check inter 2000 >> - save the state using /bin/sh -c "echo show servers state | >> /usr/bin/socat /var/run/haproxy.sock - > /var/run/haproxy/state/test" >> (this is normally done using the systemd file on reload, see initial mail) >> - reload haproxy (it still does healthchecks to port 443 while port 80 >> was expected) >> >> if you delete the statefile and reload haproxy it will start healthchecks >> for port 80 as expected >> >> -Sven >> >> >> >> >> >> >> -- >> *Van:* Baptiste >> *Verzonden:* dinsdag 3 juli 2018 11:38:14 >> *Aan:* Sven Wiltink >> *CC:* haproxy@formilux.org >> *Onderwerp:* Re: haproxy bug: healthcheck not passing after port change >> when statefile is enabled >> >> Hi Sven, >> >> Thanks a lot for your feedback! >> I'll check how we could handle this use case with the state file. >> >> Just to ensure I'm going to troubleshoot the right issue, could you >> please summarize how you trigger this issue in a
Re: haproxy bug: healthcheck not passing after port change when statefile is enabled
Hi Sven, I reviewed the whole thing and I think the support of port in state file was added for SRV records, but also for the runtime api, which allows changing the port at runtime too. I'll come back to you shortly with a fix for this behavior, currently discussing with Willy/Fred about it. (it's more complicated than moving the code """ if (port_str) srv->svc_port = port; """ a couple of lines above). Baptiste On Tue, Oct 9, 2018 at 10:52 AM Sven Wiltink wrote: > Hey Baptiste, > > > We noticed the SRV patch has been merged. That should mean that we can now > fix this issue as well. Would you be able to fix this or should we > > try to provide a patch? > > > Thanks again in advance, > > Sven > -- > *Van:* Baptiste > *Verzonden:* donderdag 12 juli 2018 14:52:24 > *Aan:* Sven Wiltink > *CC:* haproxy@formilux.org > *Onderwerp:* Re: haproxy bug: healthcheck not passing after port change > when statefile is enabled > > Hi Sven, > > Thanks for the clarification. > It's a bit more complicated than what it is supposed to be. > I think we may want to apply the port only if it has been changed at > runtime (changed by DNS SRV records). > > The status is the following: I have a pending patch which brings SRV > record information into the state file. (WIP, but last mile) > Once it has been merged, we'll be able to fix this issue (by applying the > port only when the server is being managed by an SRV record). > > Baptiste > > > On Tue, Jul 3, 2018 at 3:41 PM, Sven Wiltink wrote: > > Hey Baptiste, > > > Thank you for looking into it. > > > The bug is triggered by running haproxy with the following config: > > > global > maxconn 32000 > tune.maxrewrite 2048 > user haproxy > group haproxy > daemon > chroot /var/lib/haproxy > nbproc 1 > maxcompcpuusage 85 > spread-checks 0 > stats socket /var/run/haproxy.sock mode 600 level admin process 1 user > haproxy group haproxy > server-state-file test > server-state-base /var/run/haproxy/state > master-worker no-exit-on-failure > > defaults > load-server-state-from-file global > log global > timeout http-request 5s > timeout connect 2s > timeout client 300s > timeout server 300s > mode http > option dontlog-normal > option http-server-close > option redispatch > option log-health-checks > > listen stats > bind :1936 > bind-process 1 > mode http > stats enable > stats uri / > stats admin if TRUE > > listen banaan-443-ipv4 > bind :443 > mode tcp > server banaan-vps 127.0.0.1:443 check inter 2000 > > > - Then start haproxy (it will do healthchecks to port 443) > - change server banaan-vps 127.0.0.1:443 check inter 2000 to server > banaan-vps 127.0.0.1:80 check inter 2000 > - save the state using /bin/sh -c "echo show servers state | > /usr/bin/socat /var/run/haproxy.sock - > /var/run/haproxy/state/test" > (this is normally done using the systemd file on reload, see initial mail) > - reload haproxy (it still does healthchecks to port 443 while port 80 was > expected) > > if you delete the statefile and reload haproxy it will start healthchecks > for port 80 as expected > > -Sven > > > > > > > -- > *Van:* Baptiste > *Verzonden:* dinsdag 3 juli 2018 11:38:14 > *Aan:* Sven Wiltink > *CC:* haproxy@formilux.org > *Onderwerp:* Re: haproxy bug: healthcheck not passing after port change > when statefile is enabled > > Hi Sven, > > Thanks a lot for your feedback! > I'll check how we could handle this use case with the state file. > > Just to ensure I'm going to troubleshoot the right issue, could you please > summarize how you trigger this issue in a few simple steps? > IE: > - conf v1, server port is X > - generate server state (where port is X) > - update conf to v2, where port is Y > reload HAProxy => X is applied, while you expect to get Y instead > > Baptiste > > > > On Mon, Jun 25, 2018 at 12:55 PM, Sven Wiltink > wrote: > > Hello, > > > So we've dug a little deeper and the issue seems to be caused by the port > value in the statefile. When the target port of a server has changed > between reloads the port specified in the state file is leading. When > running tcpdump you can see the healthchecks are being performed for the > old port. After stopping haproxy and removing the statefile the healthcheck > is performed for the right
Re: haproxy bug: healthcheck not passing after port change when statefile is enabled
Hey Baptiste, We noticed the SRV patch has been merged. That should mean that we can now fix this issue as well. Would you be able to fix this or should we try to provide a patch? Thanks again in advance, Sven Van: Baptiste Verzonden: donderdag 12 juli 2018 14:52:24 Aan: Sven Wiltink CC: haproxy@formilux.org Onderwerp: Re: haproxy bug: healthcheck not passing after port change when statefile is enabled Hi Sven, Thanks for the clarification. It's a bit more complicated than what it is supposed to be. I think we may want to apply the port only if it has been changed at runtime (changed by DNS SRV records). The status is the following: I have a pending patch which brings SRV record information into the state file. (WIP, but last mile) Once it has been merged, we'll be able to fix this issue (by applying the port only when the server is being managed by an SRV record). Baptiste On Tue, Jul 3, 2018 at 3:41 PM, Sven Wiltink mailto:swilt...@transip.nl>> wrote: Hey Baptiste, Thank you for looking into it. The bug is triggered by running haproxy with the following config: global maxconn 32000 tune.maxrewrite 2048 user haproxy group haproxy daemon chroot /var/lib/haproxy nbproc 1 maxcompcpuusage 85 spread-checks 0 stats socket /var/run/haproxy.sock mode 600 level admin process 1 user haproxy group haproxy server-state-file test server-state-base /var/run/haproxy/state master-worker no-exit-on-failure defaults load-server-state-from-file global log global timeout http-request 5s timeout connect 2s timeout client 300s timeout server 300s mode http option dontlog-normal option http-server-close option redispatch option log-health-checks listen stats bind :1936 bind-process 1 mode http stats enable stats uri / stats admin if TRUE listen banaan-443-ipv4 bind :443 mode tcp server banaan-vps 127.0.0.1:443<http://127.0.0.1:443> check inter 2000 - Then start haproxy (it will do healthchecks to port 443) - change server banaan-vps 127.0.0.1:443<http://127.0.0.1:443> check inter 2000 to server banaan-vps 127.0.0.1:80<http://127.0.0.1:80> check inter 2000 - save the state using /bin/sh -c "echo show servers state | /usr/bin/socat /var/run/haproxy.sock - > /var/run/haproxy/state/test" (this is normally done using the systemd file on reload, see initial mail) - reload haproxy (it still does healthchecks to port 443 while port 80 was expected) if you delete the statefile and reload haproxy it will start healthchecks for port 80 as expected -Sven Van: Baptiste mailto:bed...@gmail.com>> Verzonden: dinsdag 3 juli 2018 11:38:14 Aan: Sven Wiltink CC: haproxy@formilux.org<mailto:haproxy@formilux.org> Onderwerp: Re: haproxy bug: healthcheck not passing after port change when statefile is enabled Hi Sven, Thanks a lot for your feedback! I'll check how we could handle this use case with the state file. Just to ensure I'm going to troubleshoot the right issue, could you please summarize how you trigger this issue in a few simple steps? IE: - conf v1, server port is X - generate server state (where port is X) - update conf to v2, where port is Y reload HAProxy => X is applied, while you expect to get Y instead Baptiste On Mon, Jun 25, 2018 at 12:55 PM, Sven Wiltink mailto:swilt...@transip.nl>> wrote: Hello, So we've dug a little deeper and the issue seems to be caused by the port value in the statefile. When the target port of a server has changed between reloads the port specified in the state file is leading. When running tcpdump you can see the healthchecks are being performed for the old port. After stopping haproxy and removing the statefile the healthcheck is performed for the right port. When manually editing the statefile to a random port the healthchecks will be performed for that port instead of the one specified by the config. The code responsible for this is line http://git.haproxy.org/?p=haproxy-1.8.git;a=blob;f=src/server.c;h=523289e3bda7ca6aa15575f1928f5298760cf582;hb=HEAD#l2931 from commit http://git.haproxy.org/?p=haproxy-1.8.git;a=commitdiff;h=3169471964fdc49963e63f68c1fd88686821a0c4. A solution would be invalidating the state when the ports don't match. -Sven Van: Sven Wiltink Verzonden: dinsdag 12 juni 2018 17:01:18 Aan: haproxy@formilux.org<mailto:haproxy@formilux.org> Onderwerp: haproxy bug: healthcheck not passing after port change when statefile is enabled Hello, There seems to be a bug in the loading of state files after a configuration change. When changing the destination port of a server the healthchecks never start passing if the state before the reload was down. This bug has been introduced a
Re: haproxy bug: healthcheck not passing after port change when statefile is enabled
Hi Sven, Thanks for the clarification. It's a bit more complicated than what it is supposed to be. I think we may want to apply the port only if it has been changed at runtime (changed by DNS SRV records). The status is the following: I have a pending patch which brings SRV record information into the state file. (WIP, but last mile) Once it has been merged, we'll be able to fix this issue (by applying the port only when the server is being managed by an SRV record). Baptiste On Tue, Jul 3, 2018 at 3:41 PM, Sven Wiltink wrote: > Hey Baptiste, > > > Thank you for looking into it. > > > The bug is triggered by running haproxy with the following config: > > > global > maxconn 32000 > tune.maxrewrite 2048 > user haproxy > group haproxy > daemon > chroot /var/lib/haproxy > nbproc 1 > maxcompcpuusage 85 > spread-checks 0 > stats socket /var/run/haproxy.sock mode 600 level admin process 1 user > haproxy group haproxy > server-state-file test > server-state-base /var/run/haproxy/state > master-worker no-exit-on-failure > > defaults > load-server-state-from-file global > log global > timeout http-request 5s > timeout connect 2s > timeout client 300s > timeout server 300s > mode http > option dontlog-normal > option http-server-close > option redispatch > option log-health-checks > > listen stats > bind :1936 > bind-process 1 > mode http > stats enable > stats uri / > stats admin if TRUE > > listen banaan-443-ipv4 > bind :443 > mode tcp > server banaan-vps 127.0.0.1:443 check inter 2000 > > > - Then start haproxy (it will do healthchecks to port 443) > - change server banaan-vps 127.0.0.1:443 check inter 2000 to server > banaan-vps 127.0.0.1:80 check inter 2000 > - save the state using /bin/sh -c "echo show servers state | > /usr/bin/socat /var/run/haproxy.sock - > /var/run/haproxy/state/test" > (this is normally done using the systemd file on reload, see initial mail) > - reload haproxy (it still does healthchecks to port 443 while port 80 was > expected) > > if you delete the statefile and reload haproxy it will start healthchecks > for port 80 as expected > > -Sven > > > > > > > ------ > *Van:* Baptiste > *Verzonden:* dinsdag 3 juli 2018 11:38:14 > *Aan:* Sven Wiltink > *CC:* haproxy@formilux.org > *Onderwerp:* Re: haproxy bug: healthcheck not passing after port change > when statefile is enabled > > Hi Sven, > > Thanks a lot for your feedback! > I'll check how we could handle this use case with the state file. > > Just to ensure I'm going to troubleshoot the right issue, could you please > summarize how you trigger this issue in a few simple steps? > IE: > - conf v1, server port is X > - generate server state (where port is X) > - update conf to v2, where port is Y > reload HAProxy => X is applied, while you expect to get Y instead > > Baptiste > > > > On Mon, Jun 25, 2018 at 12:55 PM, Sven Wiltink > wrote: > > Hello, > > > So we've dug a little deeper and the issue seems to be caused by the port > value in the statefile. When the target port of a server has changed > between reloads the port specified in the state file is leading. When > running tcpdump you can see the healthchecks are being performed for the > old port. After stopping haproxy and removing the statefile the healthcheck > is performed for the right port. When manually editing the statefile to a > random port the healthchecks will be performed for that port instead of the > one specified by the config. > > > The code responsible for this is line http://git.haproxy.org/?p=hapr > oxy-1.8.git;a=blob;f=src/server.c;h=523289e3bda7ca6aa15575f1 > 928f5298760cf582;hb=HEAD#l2931 > > from commit http://git.haproxy.org/?p=haproxy-1.8.git;a=commitdiff;h=316 > 9471964fdc49963e63f68c1fd88686821a0c4. > > > A solution would be invalidating the state when the ports don't match. > > > -Sven > > > > -- > *Van:* Sven Wiltink > *Verzonden:* dinsdag 12 juni 2018 17:01:18 > *Aan:* haproxy@formilux.org > *Onderwerp:* haproxy bug: healthcheck not passing after port change when > statefile is enabled > > Hello, > > There seems to be a bug in the loading of state files after a > configuration change. When changing the destination port of a server the > healthchecks never start passing if the state before the reload was down. > This bug has been introduced after 1.7.9 as we cannot reproduce it on > machines
Re: haproxy bug: healthcheck not passing after port change when statefile is enabled
Hey Baptiste, Thank you for looking into it. The bug is triggered by running haproxy with the following config: global maxconn 32000 tune.maxrewrite 2048 user haproxy group haproxy daemon chroot /var/lib/haproxy nbproc 1 maxcompcpuusage 85 spread-checks 0 stats socket /var/run/haproxy.sock mode 600 level admin process 1 user haproxy group haproxy server-state-file test server-state-base /var/run/haproxy/state master-worker no-exit-on-failure defaults load-server-state-from-file global log global timeout http-request 5s timeout connect 2s timeout client 300s timeout server 300s mode http option dontlog-normal option http-server-close option redispatch option log-health-checks listen stats bind :1936 bind-process 1 mode http stats enable stats uri / stats admin if TRUE listen banaan-443-ipv4 bind :443 mode tcp server banaan-vps 127.0.0.1:443 check inter 2000 - Then start haproxy (it will do healthchecks to port 443) - change server banaan-vps 127.0.0.1:443 check inter 2000 to server banaan-vps 127.0.0.1:80 check inter 2000 - save the state using /bin/sh -c "echo show servers state | /usr/bin/socat /var/run/haproxy.sock - > /var/run/haproxy/state/test" (this is normally done using the systemd file on reload, see initial mail) - reload haproxy (it still does healthchecks to port 443 while port 80 was expected) if you delete the statefile and reload haproxy it will start healthchecks for port 80 as expected -Sven Van: Baptiste Verzonden: dinsdag 3 juli 2018 11:38:14 Aan: Sven Wiltink CC: haproxy@formilux.org Onderwerp: Re: haproxy bug: healthcheck not passing after port change when statefile is enabled Hi Sven, Thanks a lot for your feedback! I'll check how we could handle this use case with the state file. Just to ensure I'm going to troubleshoot the right issue, could you please summarize how you trigger this issue in a few simple steps? IE: - conf v1, server port is X - generate server state (where port is X) - update conf to v2, where port is Y reload HAProxy => X is applied, while you expect to get Y instead Baptiste On Mon, Jun 25, 2018 at 12:55 PM, Sven Wiltink mailto:swilt...@transip.nl>> wrote: Hello, So we've dug a little deeper and the issue seems to be caused by the port value in the statefile. When the target port of a server has changed between reloads the port specified in the state file is leading. When running tcpdump you can see the healthchecks are being performed for the old port. After stopping haproxy and removing the statefile the healthcheck is performed for the right port. When manually editing the statefile to a random port the healthchecks will be performed for that port instead of the one specified by the config. The code responsible for this is line http://git.haproxy.org/?p=haproxy-1.8.git;a=blob;f=src/server.c;h=523289e3bda7ca6aa15575f1928f5298760cf582;hb=HEAD#l2931 from commit http://git.haproxy.org/?p=haproxy-1.8.git;a=commitdiff;h=3169471964fdc49963e63f68c1fd88686821a0c4. A solution would be invalidating the state when the ports don't match. -Sven Van: Sven Wiltink Verzonden: dinsdag 12 juni 2018 17:01:18 Aan: haproxy@formilux.org<mailto:haproxy@formilux.org> Onderwerp: haproxy bug: healthcheck not passing after port change when statefile is enabled Hello, There seems to be a bug in the loading of state files after a configuration change. When changing the destination port of a server the healthchecks never start passing if the state before the reload was down. This bug has been introduced after 1.7.9 as we cannot reproduce it on machines running that version of haproxy. You can use the following steps to reproduce the issue: Start with a fresh debian 9 install install socat install haproxy 1.8.9 from backports create a systemd file /etc/systemd/system/haproxy.service.d/60-haproxy-server_state.conf with the following contents: [Service] ExecStartPre=/bin/mkdir -p /var/run/haproxy/state ExecReload= ExecReload=/usr/sbin/haproxy -f ${CONFIG} -c -q $EXTRAOPTS ExecReload=/bin/sh -c "echo show servers state | /usr/bin/socat /var/run/haproxy.sock - > /var/run/haproxy/state/test" ExecReload=/bin/kill -USR2 $MAINPID create the following files: /etc/haproxy/haproxy.cfg.disabled: global maxconn 32000 tune.maxrewrite 2048 user haproxy group haproxy daemon chroot /var/lib/haproxy nbproc 1 maxcompcpuusage 85 spread-checks 0 stats socket /var/run/haproxy.sock mode 600 level admin process 1 user haproxy group haproxy server-state-file test server-state-base /var/run/haproxy/state master-worker no-exit-on-failure defaults load-server-state-from-file global log global timeout http-request
Re: haproxy bug: healthcheck not passing after port change when statefile is enabled
Hi Sven, Thanks a lot for your feedback! I'll check how we could handle this use case with the state file. Just to ensure I'm going to troubleshoot the right issue, could you please summarize how you trigger this issue in a few simple steps? IE: - conf v1, server port is X - generate server state (where port is X) - update conf to v2, where port is Y reload HAProxy => X is applied, while you expect to get Y instead Baptiste On Mon, Jun 25, 2018 at 12:55 PM, Sven Wiltink wrote: > Hello, > > > So we've dug a little deeper and the issue seems to be caused by the port > value in the statefile. When the target port of a server has changed > between reloads the port specified in the state file is leading. When > running tcpdump you can see the healthchecks are being performed for the > old port. After stopping haproxy and removing the statefile the healthcheck > is performed for the right port. When manually editing the statefile to a > random port the healthchecks will be performed for that port instead of the > one specified by the config. > > > The code responsible for this is line http://git.haproxy.org/?p= > haproxy-1.8.git;a=blob;f=src/server.c;h=523289e3bda7ca6aa15575f1928f52 > 98760cf582;hb=HEAD#l2931 > > from commit http://git.haproxy.org/?p=haproxy-1.8.git;a=commitdiff;h= > 3169471964fdc49963e63f68c1fd88686821a0c4. > > > A solution would be invalidating the state when the ports don't match. > > > -Sven > > > > -- > *Van:* Sven Wiltink > *Verzonden:* dinsdag 12 juni 2018 17:01:18 > *Aan:* haproxy@formilux.org > *Onderwerp:* haproxy bug: healthcheck not passing after port change when > statefile is enabled > > Hello, > > There seems to be a bug in the loading of state files after a > configuration change. When changing the destination port of a server the > healthchecks never start passing if the state before the reload was down. > This bug has been introduced after 1.7.9 as we cannot reproduce it on > machines running that version of haproxy. You can use the following steps > to reproduce the issue: > > Start with a fresh debian 9 install > install socat > install haproxy 1.8.9 from backports > > create a systemd file /etc/systemd/system/haproxy. > service.d/60-haproxy-server_state.conf with the following contents: > [Service] > ExecStartPre=/bin/mkdir -p /var/run/haproxy/state > ExecReload= > ExecReload=/usr/sbin/haproxy -f ${CONFIG} -c -q $EXTRAOPTS > ExecReload=/bin/sh -c "echo show servers state | /usr/bin/socat > /var/run/haproxy.sock - > /var/run/haproxy/state/test" > ExecReload=/bin/kill -USR2 $MAINPID > > create the following files: > /etc/haproxy/haproxy.cfg.disabled: > global > maxconn 32000 > tune.maxrewrite 2048 > user haproxy > group haproxy > daemon > chroot /var/lib/haproxy > nbproc 1 > maxcompcpuusage 85 > spread-checks 0 > stats socket /var/run/haproxy.sock mode 600 level admin process 1 user > haproxy group haproxy > server-state-file test > server-state-base /var/run/haproxy/state > master-worker no-exit-on-failure > > defaults > load-server-state-from-file global > log global > timeout http-request 5s > timeout connect 2s > timeout client 300s > timeout server 300s > mode http > option dontlog-normal > option http-server-close > option redispatch > option log-health-checks > > listen stats > bind :1936 > bind-process 1 > mode http > stats enable > stats uri / > stats admin if TRUE > > /etc/haproxy/haproxy.cfg.different-port: > global > maxconn 32000 > tune.maxrewrite 2048 > user haproxy > group haproxy > daemon > chroot /var/lib/haproxy > nbproc 1 > maxcompcpuusage 85 > spread-checks 0 > stats socket /var/run/haproxy.sock mode 600 level admin process 1 user > haproxy group haproxy > server-state-file test > server-state-base /var/run/haproxy/state > master-worker no-exit-on-failure > > defaults > load-server-state-from-file global > log global > timeout http-request 5s > timeout connect 2s > timeout client 300s > timeout server 300s > mode http > option dontlog-normal > option http-server-close > option redispatch > option log-health-checks > > listen stats > bind :1936 > bind-process 1 > mode http > stats enable > stats uri / > stats admin if TRUE > > listen banaan-443-ipv4 > bind :443 > mode tcp > server banaan-vps 127.0.0.1:80 check inter 2000 > listen banaan-80-ipv4 > bind :80 > mode tcp > server banaan-vps 127.0.0.1:80 check inter 2000 > > /etc/haproxy/haproxy.cfg.same-port: > global > maxconn 32000 > tune.maxrewrite 2048 > user haproxy > group haproxy > daemon > chroot /var/lib/haproxy > nbproc 1 > maxcompcpuusage 85 > spread-checks 0 > stats socket /var/run/haproxy.sock mode 600 level admin process 1 user > h
Re: haproxy bug: healthcheck not passing after port change when statefile is enabled
Hello, So we've dug a little deeper and the issue seems to be caused by the port value in the statefile. When the target port of a server has changed between reloads the port specified in the state file is leading. When running tcpdump you can see the healthchecks are being performed for the old port. After stopping haproxy and removing the statefile the healthcheck is performed for the right port. When manually editing the statefile to a random port the healthchecks will be performed for that port instead of the one specified by the config. The code responsible for this is line http://git.haproxy.org/?p=haproxy-1.8.git;a=blob;f=src/server.c;h=523289e3bda7ca6aa15575f1928f5298760cf582;hb=HEAD#l2931 from commit http://git.haproxy.org/?p=haproxy-1.8.git;a=commitdiff;h=3169471964fdc49963e63f68c1fd88686821a0c4. A solution would be invalidating the state when the ports don't match. -Sven Van: Sven Wiltink Verzonden: dinsdag 12 juni 2018 17:01:18 Aan: haproxy@formilux.org Onderwerp: haproxy bug: healthcheck not passing after port change when statefile is enabled Hello, There seems to be a bug in the loading of state files after a configuration change. When changing the destination port of a server the healthchecks never start passing if the state before the reload was down. This bug has been introduced after 1.7.9 as we cannot reproduce it on machines running that version of haproxy. You can use the following steps to reproduce the issue: Start with a fresh debian 9 install install socat install haproxy 1.8.9 from backports create a systemd file /etc/systemd/system/haproxy.service.d/60-haproxy-server_state.conf with the following contents: [Service] ExecStartPre=/bin/mkdir -p /var/run/haproxy/state ExecReload= ExecReload=/usr/sbin/haproxy -f ${CONFIG} -c -q $EXTRAOPTS ExecReload=/bin/sh -c "echo show servers state | /usr/bin/socat /var/run/haproxy.sock - > /var/run/haproxy/state/test" ExecReload=/bin/kill -USR2 $MAINPID create the following files: /etc/haproxy/haproxy.cfg.disabled: global maxconn 32000 tune.maxrewrite 2048 user haproxy group haproxy daemon chroot /var/lib/haproxy nbproc 1 maxcompcpuusage 85 spread-checks 0 stats socket /var/run/haproxy.sock mode 600 level admin process 1 user haproxy group haproxy server-state-file test server-state-base /var/run/haproxy/state master-worker no-exit-on-failure defaults load-server-state-from-file global log global timeout http-request 5s timeout connect 2s timeout client 300s timeout server 300s mode http option dontlog-normal option http-server-close option redispatch option log-health-checks listen stats bind :1936 bind-process 1 mode http stats enable stats uri / stats admin if TRUE /etc/haproxy/haproxy.cfg.different-port: global maxconn 32000 tune.maxrewrite 2048 user haproxy group haproxy daemon chroot /var/lib/haproxy nbproc 1 maxcompcpuusage 85 spread-checks 0 stats socket /var/run/haproxy.sock mode 600 level admin process 1 user haproxy group haproxy server-state-file test server-state-base /var/run/haproxy/state master-worker no-exit-on-failure defaults load-server-state-from-file global log global timeout http-request 5s timeout connect 2s timeout client 300s timeout server 300s mode http option dontlog-normal option http-server-close option redispatch option log-health-checks listen stats bind :1936 bind-process 1 mode http stats enable stats uri / stats admin if TRUE listen banaan-443-ipv4 bind :443 mode tcp server banaan-vps 127.0.0.1:80 check inter 2000 listen banaan-80-ipv4 bind :80 mode tcp server banaan-vps 127.0.0.1:80 check inter 2000 /etc/haproxy/haproxy.cfg.same-port: global maxconn 32000 tune.maxrewrite 2048 user haproxy group haproxy daemon chroot /var/lib/haproxy nbproc 1 maxcompcpuusage 85 spread-checks 0 stats socket /var/run/haproxy.sock mode 600 level admin process 1 user haproxy group haproxy server-state-file test server-state-base /var/run/haproxy/state master-worker no-exit-on-failure defaults load-server-state-from-file global log global timeout http-request 5s timeout connect 2s timeout client 300s timeout server 300s mode http option dontlog-normal option http-server-close option redispatch option log-health-checks listen stats bind :1936 bind-process 1 mode http stats enable stats uri / stats admin if TRUE listen banaan-443-ipv4 bind :443 mode tcp server banaan-vps 127.0.0.1:443 check inter 2000 listen banaan-80-ipv4 bind :80 mode tcp server banaan-vps 127.0.0.1:80 check inter 2000 start a netcat p
Re: haproxy bug: healthcheck not passing after port change when statefile is enabled
Sven, Am 12.06.2018 um 17:01 schrieb Sven Wiltink: > create a systemd file > /etc/systemd/system/haproxy.service.d/60-haproxy-server_state.conf with the > following contents: > [Service] > ExecStartPre=/bin/mkdir -p /var/run/haproxy/state > ExecReload= > ExecReload=/usr/sbin/haproxy -f ${CONFIG} -c -q $EXTRAOPTS > ExecReload=/bin/sh -c "echo show servers state | /usr/bin/socat > /var/run/haproxy.sock - > /var/run/haproxy/state/test" > ExecReload=/bin/kill -USR2 $MAINPID While this would not have an effect on your issue I suggest specifying RuntimeDirectory to save the manual `mkdir` and manual clean up: > # /lib/systemd/system/haproxy.service.d/state.conf > [Service] > RuntimeDirectory=haproxy > ExecReload= > ExecReload=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS > ExecReload=/bin/sh -c "echo show servers state |nc -U > /var/run/haproxy/admin.sock > /run/haproxy/global-state" > ExecReload=/bin/kill -USR2 $MAINPID The RuntimeDirectory would automatically be cleaned on `restart` / `stop`, but not on a `reload`. If this is not wanted take a look at `RuntimeDirectoryPreserve` (https://www.freedesktop.org/software/systemd/man/systemd.exec.html#RuntimeDirectoryPreserve=). Best regards Tim Düsterhus