Re: tcp-check expect with exclamation mark
2018-06-22 1:13 GMT+10:00 Baptiste : > > > On Thu, Jun 21, 2018 at 8:53 AM, Igor Cicimov com> wrote: > >> Hi Dmitriy, >> >> On Thu, Jun 21, 2018 at 12:45 PM, Dmitriy Kuzmin >> wrote: >> >>> Greetings >>> >>> I’m using haproxy to load balance readonly queries between redis slaves. >>> I want to use health check system to exclude slaves from load balancing, >>> that are in a process of sync with master. >>> The idea is to look for a string “master_sync_in_progress:1” in response >>> to “info replication”. >>> >> >> Is this _exactly_ what is returned? >> >> >>> If this string is found then backend should be marked as down. >>> >>> I’m trying to use “tcp-check expect ! string” (with exclamation mark >>> [!]) to get this working, but backends are permanently down regardless of >>> sync status. >>> During sync (slave’s response contains “master_sync_in_progress:1”) >>> health check status is “L7RSP,TCPCHK matched unwanted content >>> 'master_sync_in_progress:1' at step 5”. >>> However, when slave’s response contains “master_sync_in_progress:0” >>> (sync finished) health check status is “L7TOUT, at step 5 of tcp-check >>> (expect string 'master_sync_in_progress:1’)”. >>> >>> Does negation with exclamation mark (!) work with tcp-check at all? >>> >>> I’ve observed this behaviour in haproxy 1.5.18, 1.7.11 and 1.8.9 >>> >>> >>> Example configuration: >>> >>> global >>> log 127.0.0.1 local2 >>> chroot/var/lib/haproxy >>> pidfile /var/run/haproxy.pid >>> maxconn 4000 >>> user haproxy >>> group haproxy >>> daemon >>> >>> defaults >>> modetcp >>> log global >>> option tcplog >>> option dontlognull >>> option redispatch >>> retries 3 >>> timeout http-request10s >>> timeout queue 1m >>> timeout connect 10s >>> timeout client 1m >>> timeout server 1m >>> timeout http-keep-alive 10s >>> timeout check 10s >>> maxconn 3000 >>> >>> frontend redis_reads_ft >>> bind/var/run/haproxy/redis_reads >>> use_backend redis_reads_bk >>> >>> backend redis_reads_bk >>> option log-health-checks >>> balance roundrobin >>> >>> optiontcp-check >>> tcp-check connect >>> tcp-check send PING\r\n >>> tcp-check expect string +PONG >>> tcp-check send info\ replication\r\n >>> tcp-check expect ! string >>> >>> master_sync_in_progress:1 >>> >> >> Try using *rstring* intead of *string*. I that fails too try escaping >> the column like "master_sync_in_progress\:1" >> >> tcp-check send QUIT\r\n >>> tcp-check expect string +OK >>> >>> server sc-redis1_63811 10.10.68.61:63811 check >>> server sc-redis1_63812 10.10.68.61:63812 check >>> server sc-redis1_63813 10.10.68.61:63813 check >>> >>> >>> Best regards, >>> Dmitriy Kuzmin >>> >> >> > > I'm not sure what string you're trying to match. Could you paste the > output of "info replication" somewhere on pastebin or gist? > > Baptiste > Greetings Thank you for your reply Here is full output from "healthy" backend: https://pastebin.com/in9yH4e4 Here is full output from "unhealthy" backend: https://pastebin.com/FSPaiKW0 I'm trying to match a string "master_sync_in_progress:1" which is contained only in "unhealthy" backend response. With this rules (mention the exclamation mark): tcp-check connect tcp-check send info\ replication\r\n tcp-check expect ! string master_sync_in_progress:1 i expect that only backends, whose response contains "master_sync_in_progress:1" will be marked as DOWN. However, both healthy and unhealthy backends are marked as DOWN. Unhealthy backends correctly marked as DOWN because there is a match: "matched unwanted content 'master_sync_in_progress:1 at step 5" But healthy backends are marked as DOWN too. Because of “L7TOUT, at step 5 of tcp-check (expect string 'master_sync_in_progress:1’)”. And i'm trying to figure out why is that. Any guess? Best regards, Dmitriy Kuzmin
Re: tcp-check expect with exclamation mark
On Thu, Jun 21, 2018 at 8:53 AM, Igor Cicimov < ig...@encompasscorporation.com> wrote: > Hi Dmitriy, > > On Thu, Jun 21, 2018 at 12:45 PM, Dmitriy Kuzmin > wrote: > >> Greetings >> >> I’m using haproxy to load balance readonly queries between redis slaves. >> I want to use health check system to exclude slaves from load balancing, >> that are in a process of sync with master. >> The idea is to look for a string “master_sync_in_progress:1” in response >> to “info replication”. >> > > Is this _exactly_ what is returned? > > >> If this string is found then backend should be marked as down. >> >> I’m trying to use “tcp-check expect ! string” (with exclamation mark [!]) >> to get this working, but backends are permanently down regardless of sync >> status. >> During sync (slave’s response contains “master_sync_in_progress:1”) >> health check status is “L7RSP,TCPCHK matched unwanted content >> 'master_sync_in_progress:1' at step 5”. >> However, when slave’s response contains “master_sync_in_progress:0” (sync >> finished) health check status is “L7TOUT, at step 5 of tcp-check (expect >> string 'master_sync_in_progress:1’)”. >> >> Does negation with exclamation mark (!) work with tcp-check at all? >> >> I’ve observed this behaviour in haproxy 1.5.18, 1.7.11 and 1.8.9 >> >> >> Example configuration: >> >> global >> log 127.0.0.1 local2 >> chroot/var/lib/haproxy >> pidfile /var/run/haproxy.pid >> maxconn 4000 >> user haproxy >> group haproxy >> daemon >> >> defaults >> modetcp >> log global >> option tcplog >> option dontlognull >> option redispatch >> retries 3 >> timeout http-request10s >> timeout queue 1m >> timeout connect 10s >> timeout client 1m >> timeout server 1m >> timeout http-keep-alive 10s >> timeout check 10s >> maxconn 3000 >> >> frontend redis_reads_ft >> bind/var/run/haproxy/redis_reads >> use_backend redis_reads_bk >> >> backend redis_reads_bk >> option log-health-checks >> balance roundrobin >> >> optiontcp-check >> tcp-check connect >> tcp-check send PING\r\n >> tcp-check expect string +PONG >> tcp-check send info\ replication\r\n >> tcp-check expect ! string >> >> master_sync_in_progress:1 >> > > Try using *rstring* intead of *string*. I that fails too try escaping > the column like "master_sync_in_progress\:1" > > tcp-check send QUIT\r\n >> tcp-check expect string +OK >> >> server sc-redis1_63811 10.10.68.61:63811 check >> server sc-redis1_63812 10.10.68.61:63812 check >> server sc-redis1_63813 10.10.68.61:63813 check >> >> >> Best regards, >> Dmitriy Kuzmin >> > > I'm not sure what string you're trying to match. Could you paste the output of "info replication" somewhere on pastebin or gist? Baptiste
Re: tcp-check expect with exclamation mark
Hi Dmitriy, On Thu, Jun 21, 2018 at 12:45 PM, Dmitriy Kuzmin wrote: > Greetings > > I’m using haproxy to load balance readonly queries between redis slaves. > I want to use health check system to exclude slaves from load balancing, > that are in a process of sync with master. > The idea is to look for a string “master_sync_in_progress:1” in response > to “info replication”. > Is this _exactly_ what is returned? > If this string is found then backend should be marked as down. > > I’m trying to use “tcp-check expect ! string” (with exclamation mark [!]) > to get this working, but backends are permanently down regardless of sync > status. > During sync (slave’s response contains “master_sync_in_progress:1”) health > check status is “L7RSP,TCPCHK matched unwanted content > 'master_sync_in_progress:1' at step 5”. > However, when slave’s response contains “master_sync_in_progress:0” (sync > finished) health check status is “L7TOUT, at step 5 of tcp-check (expect > string 'master_sync_in_progress:1’)”. > > Does negation with exclamation mark (!) work with tcp-check at all? > > I’ve observed this behaviour in haproxy 1.5.18, 1.7.11 and 1.8.9 > > > Example configuration: > > global > log 127.0.0.1 local2 > chroot/var/lib/haproxy > pidfile /var/run/haproxy.pid > maxconn 4000 > user haproxy > group haproxy > daemon > > defaults > modetcp > log global > option tcplog > option dontlognull > option redispatch > retries 3 > timeout http-request10s > timeout queue 1m > timeout connect 10s > timeout client 1m > timeout server 1m > timeout http-keep-alive 10s > timeout check 10s > maxconn 3000 > > frontend redis_reads_ft > bind/var/run/haproxy/redis_reads > use_backend redis_reads_bk > > backend redis_reads_bk > option log-health-checks > balance roundrobin > > optiontcp-check > tcp-check connect > tcp-check send PING\r\n > tcp-check expect string +PONG > tcp-check send info\ replication\r\n > tcp-check expect ! string > > master_sync_in_progress:1 > Try using *rstring* intead of *string*. I that fails too try escaping the column like " master_sync_in_progress\:1" tcp-check send QUIT\r\n > tcp-check expect string +OK > > server sc-redis1_63811 10.10.68.61:63811 check > server sc-redis1_63812 10.10.68.61:63812 check > server sc-redis1_63813 10.10.68.61:63813 check > > > Best regards, > Dmitriy Kuzmin >
tcp-check expect with exclamation mark
Greetings I’m using haproxy to load balance readonly queries between redis slaves. I want to use health check system to exclude slaves from load balancing, that are in a process of sync with master. The idea is to look for a string “master_sync_in_progress:1” in response to “info replication”. If this string is found then backend should be marked as down. I’m trying to use “tcp-check expect ! string” (with exclamation mark [!]) to get this working, but backends are permanently down regardless of sync status. During sync (slave’s response contains “master_sync_in_progress:1”) health check status is “L7RSP,TCPCHK matched unwanted content 'master_sync_in_progress:1' at step 5”. However, when slave’s response contains “master_sync_in_progress:0” (sync finished) health check status is “L7TOUT, at step 5 of tcp-check (expect string 'master_sync_in_progress:1’)”. Does negation with exclamation mark (!) work with tcp-check at all? I’ve observed this behaviour in haproxy 1.5.18, 1.7.11 and 1.8.9 Example configuration: global log 127.0.0.1 local2 chroot/var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon defaults modetcp log global option tcplog option dontlognull option redispatch retries 3 timeout http-request10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000 frontend redis_reads_ft bind/var/run/haproxy/redis_reads use_backend redis_reads_bk backend redis_reads_bk option log-health-checks balance roundrobin optiontcp-check tcp-check connect tcp-check send PING\r\n tcp-check expect string +PONG tcp-check send info\ replication\r\n tcp-check expect ! string master_sync_in_progress:1 tcp-check send QUIT\r\n tcp-check expect string +OK server sc-redis1_63811 10.10.68.61:63811 check server sc-redis1_63812 10.10.68.61:63812 check server sc-redis1_63813 10.10.68.61:63813 check Best regards, Dmitriy Kuzmin
