BUG: Extra spaces inserted into the haproxy.cfg file cause haproxy reload with saved state to seg fault haproxy.
WORKAROUND: Remove the existing state file OR remove save state from config. REPRODUCE: - Enable save state across reloads - Reload and save state file - Insert extra space before "weight". - Attempt to reload or validate the new config and haproxy will segv. DESCRIPTION Our haproxy config is comprised of a bunch of puppet magic. Someone made a change that included an additional space in nearly every server line of our config, which has almost 1300 servers: - server foo3001.xx.box.net 10.16.26.2:8080 weight 16 check port 9086 inter 10000 - server foo3002.xx.box.net 10.16.26.3:8080 weight 16 check port 9086 inter 10000 - server foo3003.xx.box.net 10.16.26.4:8080 weight 32 check port 9086 inter 10000 - server foo3004.xx.box.net 10.16.26.5:8080 weight 16 check port 9086 inter 10000 + server foo3001.xx.box.net 10.16.26.2:8080 weight 16 check port 9086 inter 10000 + server foo3002.xx.box.net 10.16.26.3:8080 weight 16 check port 9086 inter 10000 + server foo3003.xx.box.net 10.16.26.4:8080 weight 32 check port 9086 inter 10000 + server foo3004.xx.box.net 10.16.26.5:8080 weight 16 check port 9086 inter 10000 We are testing save server state on some of some of our hosts. This change causes reloads to fail with a segmentation fault. If I remove the state saving stuff from the config, the segv goes away: % diff ~/haproxy*cfg 15,16c15,16 < # stats socket /var/run/haproxy.sock mode 666 level admin < # server-state-file /var/lib/haproxy/haproxy-server-state --- > stats socket /var/run/haproxy.sock mode 666 level admin > server-state-file /var/lib/haproxy/haproxy-server-state 20c20 < # load-server-state-from-file global --- > load-server-state-from-file global % /usr/sbin/haproxy -c -f ~/haproxy-test.cfg Configuration file is valid % If I remove the state file, the config check is valid: % rm /var/lib/haproxy/haproxy-server-state % /usr/sbin/haproxy -c -f ~/haproxy.cfg [WARNING] 149/180256 (140127) : stats socket will not work as expected in multi-process mode (nbproc > 1), you should force process binding globally using 'stats bind-process' or per socket using the 'process' attribute. [WARNING] 149/180256 (140127) : Can't open server state file '/var/lib/haproxy/haproxy-server-state': No such file or directory [WARNING] 149/180256 (140127) : Can't open server state file '/var/lib/haproxy/haproxy-server-state': No such file or directory [WARNING] 149/180256 (140127) : Can't open server state file '/var/lib/haproxy/haproxy-server-state': No such file or directory [WARNING] 149/180256 (140127) : Can't open server state file '/var/lib/haproxy/haproxy-server-state': No such file or directory [WARNING] 149/180256 (140127) : Can't open server state file '/var/lib/haproxy/haproxy-server-state': No such file or directory [WARNING] 149/180256 (140127) : Can't open server state file '/var/lib/haproxy/haproxy-server-state': No such file or directory [WARNING] 149/180256 (140127) : Can't open server state file '/var/lib/haproxy/haproxy-server-state': No such file or directory [WARNING] 149/180256 (140127) : Can't open server state file '/var/lib/haproxy/haproxy-server-state': No such file or directory [WARNING] 149/180256 (140127) : Can't open server state file '/var/lib/haproxy/haproxy-server-state': No such file or directory [WARNING] 149/180256 (140127) : Can't open server state file '/var/lib/haproxy/haproxy-server-state': No such file or directory [WARNING] 149/180256 (140127) : Can't open server state file '/var/lib/haproxy/haproxy-server-state': No such file or directory [WARNING] 149/180256 (140127) : Can't open server state file '/var/lib/haproxy/haproxy-server-state': No such file or directory [WARNING] 149/180256 (140127) : Can't open server state file '/var/lib/haproxy/haproxy-server-state': No such file or directory [WARNING] 149/180256 (140127) : Can't open server state file '/var/lib/haproxy/haproxy-server-state': No such file or directory [WARNING] 149/180256 (140127) : Can't open server state file '/var/lib/haproxy/haproxy-server-state': No such file or directory [WARNING] 149/180256 (140127) : Can't open server state file '/var/lib/haproxy/haproxy-server-state': No such file or directory [WARNING] 149/180256 (140127) : Can't open server state file '/var/lib/haproxy/haproxy-server-state': No such file or directory [WARNING] 149/180256 (140127) : Can't open server state file '/var/lib/haproxy/haproxy-server-state': No such file or directory [WARNING] 149/180256 (140127) : Can't open server state file '/var/lib/haproxy/haproxy-server-state': No such file or directory [WARNING] 149/180256 (140127) : Can't open server state file '/var/lib/haproxy/haproxy-server-state': No such file or directory [WARNING] 149/180256 (140127) : Can't open server state file '/var/lib/haproxy/haproxy-server-state': No such file or directory [WARNING] 149/180256 (140127) : Can't open server state file '/var/lib/haproxy/haproxy-server-state': No such file or directory [WARNING] 149/180256 (140127) : Can't open server state file '/var/lib/haproxy/haproxy-server-state': No such file or directory [WARNING] 149/180256 (140127) : Can't open server state file '/var/lib/haproxy/haproxy-server-state': No such file or directory Configuration file is valid % ------------------------------------------------------------------------------------------- >From /etc/haproxy/haproxy.cfg: global stats socket /var/run/haproxy.sock mode 666 level admin server-state-file /var/lib/haproxy/haproxy-server-state defaults log global load-server-state-from-file global ------------------------------------------------------------------------------------------- % /usr/sbin/haproxy -vv HA-Proxy version 1.7.5 2017/04/03 Copyright 2000-2017 Willy Tarreau <[email protected]> Build options : TARGET = linux2628 CPU = generic CC = gcc CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement OPTIONS = USE_LINUX_TPROXY=1 USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Encrypted password support via crypt(3): yes Built with zlib version : 1.2.3 Running on zlib version : 1.2.3 Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip") Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 7.8 2008-09-05 Running on PCRE version : 7.8 2008-09-05 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built without Lua support Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. Available filters : [SPOE] spoe [TRACE] trace [COMP] compression % % /usr/sbin/haproxy -c -V -f ~/haproxy.cfg [WARNING] 149/175354 (114894) : stats socket will not work as expected in multi-process mode (nbproc > 1), you should force process binding globally using 'stats bind-process' or per socket using the 'process' attribute. [WARNING] 149/175354 (114894) : Server http_w_c1_6000/c13001.xx.box.net is DOWN, changed from server-state after a reload. 230 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server http_w_c1_6000/c13002.xx.box.net is DOWN, changed from server-state after a reload. 229 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server http_w_c1_6000/c13003.xx.box.net is DOWN, changed from server-state after a reload. 228 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server http_w_c1_6000/c13004.xx.box.net is DOWN, changed from server-state after a reload. 227 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server http_w_c1_6000/c13032.xx.box.net is DOWN, changed from server-state after a reload. 226 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server http_w_c1_6000/c13083.xx.box.net is DOWN, changed from server-state after a reload. 225 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server http_w_c1_6000/c13105.xx.box.net is DOWN, changed from server-state after a reload. 224 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server http_w_c1_6000/c13122.xx.box.net is DOWN, changed from server-state after a reload. 223 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server http_w_c1_6000/c13189.xx.box.net is DOWN, changed from server-state after a reload. 222 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server http_w_c1_6000/c13214.xx.box.net is DOWN, changed from server-state after a reload. 221 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server http_w_c1_6000/c13288.xx.box.net is DOWN, changed from server-state after a reload. 220 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server http_w_c1_6000/c13319.xx.box.net is DOWN, changed from server-state after a reload. 219 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server http_w_c1_6000/c13321.xx.box.net is DOWN, changed from server-state after a reload. 218 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server https_d_6014/d13006.xx.box.net is DOWN, changed from server-state after a reload. 30 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server https_d_6014/d13008.xx.box.net is DOWN, changed from server-state after a reload. 29 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server https_d_6014/d13011.xx.box.net is DOWN, changed from server-state after a reload. 28 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server https_d_6014/d13015.xx.box.net is DOWN, changed from server-state after a reload. 27 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server https_d_6014/d13024.xx.box.net is DOWN, changed from server-state after a reload. 26 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server https_d_6014/d13031.xx.box.net is DOWN, changed from server-state after a reload. 25 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server https_d_6014/d13032.xx.box.net is DOWN, changed from server-state after a reload. 24 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server https_d_6014/d13035.xx.box.net is DOWN, changed from server-state after a reload. 23 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server https_ms_6005/c13001.xx.box.net is DOWN, changed from server-state after a reload. 230 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server https_ms_6005/c13002.xx.box.net is DOWN, changed from server-state after a reload. 229 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server https_ms_6005/c13003.xx.box.net is DOWN, changed from server-state after a reload. 228 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server https_ms_6005/c13004.xx.box.net is DOWN, changed from server-state after a reload. 227 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server https_ms_6005/c13032.xx.box.net is DOWN, changed from server-state after a reload. 226 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server https_ms_6005/c13083.xx.box.net is DOWN, changed from server-state after a reload. 225 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server https_ms_6005/c13105.xx.box.net is DOWN, changed from server-state after a reload. 224 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server https_ms_6005/c13122.xx.box.net is DOWN, changed from server-state after a reload. 223 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server https_ms_6005/c13189.xx.box.net is DOWN, changed from server-state after a reload. 222 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server https_ms_6005/c13214.xx.box.net is DOWN, changed from server-state after a reload. 221 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server https_ms_6005/c13288.xx.box.net is DOWN, changed from server-state after a reload. 220 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server https_ms_6005/c13319.xx.box.net is DOWN, changed from server-state after a reload. 219 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server https_ms_6005/c13321.xx.box.net is DOWN, changed from server-state after a reload. 218 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server https_u_u_6003/u3050.xx.box.net is DOWN, changed from server-state after a reload. 113 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server https_u_u_6003/u3053.xx.box.net is DOWN, changed from server-state after a reload. 112 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server https_u_u_6003/u3063.xx.box.net is DOWN, changed from server-state after a reload. 111 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server https_u_u_6003/u3070.xx.box.net is DOWN, changed from server-state after a reload. 110 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server https_u_u_6003/u3100.xx.box.net is DOWN, changed from server-state after a reload. 109 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server https_u_u_6003/u3120.xx.box.net is DOWN, changed from server-state after a reload. 108 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : Server https_u_u_6003/u3121.xx.box.net is DOWN, changed from server-state after a reload. 107 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : In backend 'https_u_u_6003' (id: 17): server name mismatch: from server state file: 'u3126.xx.box.net', from running config 'u3127.xx.box.net' [WARNING] 149/175354 (114894) : Server https_u_u_6003/u3127.xx.box.net is DOWN, changed from server-state after a reload. 106 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 149/175354 (114894) : In backend 'https_u_u_6003' (id: '17'): server ID mismatch: from server state file: '98', from running config 97 Segmentation fault %
