Hi, What consul version are you using? I'm facing the same issue in my consul lab. That said, it seems to be a bug in consul, not able to serve too many SRV records over UDP. I even triggered a consul crash (using 1.0.5 version). I'm still investigating this issue and will come back to you as soon as I have more reliable information.
Note: please ensure the number of server created by server-template directive (5 in your case) is above the expected number of server available in your service. Baptiste On Thu, Feb 8, 2018 at 12:32 AM, Чепайкин Михаил <[email protected]> wrote: > Hi > > I’ve changed configuration as you suggested: > > backend tsdb_backend_query > server-template tsdb_query 5 > _mfm-monitor-opentsdb._tcp.service.mfmconsul:4242 check resolvers dns inter > 1000 > > Logs are kinda different - backend servers now go UP and DOWN, but seems > the same - ip addresses changing in the same way: > > time="2018-02-08T02:12:53+03:00" level=info msg="[WARNING] 038/021253 (18208) > : Server tsdb_backend_query/tsdb_query1 is going DOWN for maintenance (No IP > for server ). 2 active and 0 backup servers left. 0 sessions active, 0 > requeued, 0 remaining in queue." job=mfm-monitor-haproxy pid=18208 > time="2018-02-08T02:12:53+03:00" level=info msg="[WARNING] 038/021253 (18208) > : tsdb_backend_query/tsdb_query1 changed its IP from 10.182.161.223 to > 10.182.161.211 by DNS cache." job=mfm-monitor-haproxy pid=18208 > time="2018-02-08T02:12:53+03:00" level=info msg="[WARNING] 038/021253 (18208) > : Server tsdb_backend_query/tsdb_query1 administratively READY thanks to > valid DNS answer." job=mfm-monitor-haproxy pid=18208 > time="2018-02-08T02:12:53+03:00" level=info msg="[WARNING] 038/021253 (18208) > : Server tsdb_backend_query/tsdb_query1 ('0ab6a1d3.addr.dc1.mfmconsul') is > UP/READY (resolves again)." job=mfm-monitor-haproxy pid=18208 > time="2018-02-08T02:12:55+03:00" level=info msg="[WARNING] 038/021255 (18208) > : Server tsdb_backend_query/tsdb_query3 is going DOWN for maintenance (No IP > for server ). 2 active and 0 backup servers left. 0 sessions active, 0 > requeued, 0 remaining in queue." job=mfm-monitor-haproxy pid=18208 > time="2018-02-08T02:12:55+03:00" level=info msg="[WARNING] 038/021255 (18208) > : tsdb_backend_query/tsdb_query3 changed its IP from 10.182.161.98 to > 10.182.161.223 by DNS cache." job=mfm-monitor-haproxy pid=18208 > time="2018-02-08T02:12:55+03:00" level=info msg="[WARNING] 038/021255 (18208) > : Server tsdb_backend_query/tsdb_query3 administratively READY thanks to > valid DNS answer." job=mfm-monitor-haproxy pid=18208 > time="2018-02-08T02:12:55+03:00" level=info msg="[WARNING] 038/021255 (18208) > : Server tsdb_backend_query/tsdb_query3 ('0ab6a1df.addr.dc1.mfmconsul') is > UP/READY (resolves again)." job=mfm-monitor-haproxy pid=18208 > time="2018-02-08T02:12:57+03:00" level=info msg="[WARNING] 038/021257 (18208) > : Server tsdb_backend_query/tsdb_query3 is going DOWN for maintenance (No IP > for server ). 2 active and 0 backup servers left. 0 sessions active, 0 > requeued, 0 remaining in queue." job=mfm-monitor-haproxy pid=18208 > time="2018-02-08T02:12:57+03:00" level=info msg="[WARNING] 038/021257 (18208) > : tsdb_backend_query/tsdb_query3 changed its IP from 10.182.161.223 to > 10.182.161.98 by DNS cache." job=mfm-monitor-haproxy pid=18208 > time="2018-02-08T02:12:57+03:00" level=info msg="[WARNING] 038/021257 (18208) > : Server tsdb_backend_query/tsdb_query3 administratively READY thanks to > valid DNS answer." job=mfm-monitor-haproxy pid=18208 > time="2018-02-08T02:12:57+03:00" level=info msg="[WARNING] 038/021257 (18208) > : Server tsdb_backend_query/tsdb_query3 ('0ab6a162.addr.dc1.mfmconsul') is > UP/READY (resolves again)." job=mfm-monitor-haproxy pid=18208 > time="2018-02-08T02:13:01+03:00" level=info msg="[WARNING] 038/021301 (18208) > : Server tsdb_backend_query/tsdb_query1 is going DOWN for maintenance (No IP > for server ). 2 active and 0 backup servers left. 0 sessions active, 0 > requeued, 0 remaining in queue." job=mfm-monitor-haproxy pid=18208 > time="2018-02-08T02:13:01+03:00" level=info msg="[WARNING] 038/021301 (18208) > : tsdb_backend_query/tsdb_query1 changed its IP from 10.182.161.211 to > 10.182.161.223 by DNS cache." job=mfm-monitor-haproxy pid=18208 > time="2018-02-08T02:13:01+03:00" level=info msg="[WARNING] 038/021301 (18208) > : Server tsdb_backend_query/tsdb_query1 administratively READY thanks to > valid DNS answer." job=mfm-monitor-haproxy pid=18208 > time="2018-02-08T02:13:01+03:00" level=info msg="[WARNING] 038/021301 (18208) > : Server tsdb_backend_query/tsdb_query1 ('0ab6a1df.addr.dc1.mfmconsul') is > UP/READY (resolves again)." job=mfm-monitor-haproxy pid=18208 > time="2018-02-08T02:13:05+03:00" level=info msg="[WARNING] 038/021305 (18208) > : Server tsdb_backend_query/tsdb_query2 is going DOWN for maintenance (No IP > for server ). 2 active and 0 backup servers left. 0 sessions active, 0 > requeued, 0 remaining in queue." job=mfm-monitor-haproxy pid=18208 > time="2018-02-08T02:13:05+03:00" level=info msg="[WARNING] 038/021305 (18208) > : tsdb_backend_query/tsdb_query2 changed its IP from 10.182.161.163 to > 10.182.161.211 by DNS cache." job=mfm-monitor-haproxy pid=18208 > > Any thoughts? > > On 8 February 2018 at 01:25, Baptiste <[email protected]> wrote: > > Hi >> >> You're not using SRV records and that may be the root cause of your issue. >> Please try something like this: >> >> backend tsdb_backend_query >> server-template tsdb_query 5 >> _mfm-monitor-opentsdb._tcp.service.mfmconsul:4242 check resolvers dns inter >> 1000 >> >> if "mfm-monitor-opentsdb" is your service name in consul. >> >> Baptiste >> >> >> >> On Wed, Feb 7, 2018 at 2:52 PM, Чепайкин Михаил <[email protected]> >> wrote: >> >>> Hi! >>> >>> I have a Consul as service discovery tool and HAProxy as load balancer. >>> >>> In Consul registered a service running on a number of servers, and this >>> service can be scaled by adding and removing nodes and by moving nodes from >>> one server to another. >>> >>> Consul has DNS service which randomizes responses for services like that: >>> >>> [bux] michep@bux:~$ dig +short mfm-monitor-opentsdb.service.mfmconsul >>> 10.182.161.239 >>> 10.182.161.152 >>> 10.182.161.240 >>> 10.182.161.92 >>> [bux] michep@bux:~$ dig +short mfm-monitor-opentsdb.service.mfmconsul >>> 10.182.161.92 >>> 10.182.161.152 >>> 10.182.161.240 >>> 10.182.161.239 >>> >>> In HAProxy 1.8.3 im using server-template configuration, like that: >>> >>> resolvers dns >>> nameserver dns1 ${HAPROXY_NAMESERVER} >>> hold valid 2s >>> >>> backend tsdb_backend_query >>> server-template tsdb_query 5 mfm-monitor-opentsdb.service.mfmconsul:4242 >>> check resolvers dns inter 1000 >>> >>> And in that case I get alot of warinings in haproxy log: >>> >>> time="2018-02-02T15:44:32+03:00" level=info msg="[WARNING] 032/154432 >>> (32983) : tsdb_backend_query/tsdb_query1 changed its IP from 10.182.161.240 >>> to 10.182.161.239 by DNS cache." job=mfm-monitor-haproxy pid=32983 >>> time="2018-02-02T15:44:42+03:00" level=info msg="[WARNING] 032/154442 >>> (32983) : tsdb_backend_query/tsdb_query1 changed its IP from 10.182.161.239 >>> to 10.182.161.240 by DNS cache." job=mfm-monitor-haproxy pid=32983 >>> time="2018-02-02T15:44:46+03:00" level=info msg="[WARNING] 032/154446 >>> (32983) : tsdb_backend_query/tsdb_query3 changed its IP from 10.182.161.152 >>> to 10.182.161.239 by DNS cache." job=mfm-monitor-haproxy pid=32983 >>> time="2018-02-02T15:44:50+03:00" level=info msg="[WARNING] 032/154450 >>> (32983) : tsdb_backend_query/tsdb_query2 changed its IP from 10.182.161.92 >>> to 10.182.161.152 by DNS cache." job=mfm-monitor-haproxy pid=32983 >>> time="2018-02-02T15:44:52+03:00" level=info msg="[WARNING] 032/154452 >>> (32983) : tsdb_backend_query/tsdb_query3 changed its IP from 10.182.161.239 >>> to 10.182.161.92 by DNS cache." job=mfm-monitor-haproxy pid=32983 >>> time="2018-02-02T15:44:56+03:00" level=info msg="[WARNING] 032/154456 >>> (32983) : tsdb_backend_query/tsdb_query1 changed its IP from 10.182.161.240 >>> to 10.182.161.239 by DNS cache." job=mfm-monitor-haproxy pid=32983 >>> time="2018-02-02T15:45:00+03:00" level=info msg="[WARNING] 032/154500 >>> (32983) : tsdb_backend_query/tsdb_query3 changed its IP from 10.182.161.92 >>> to 10.182.161.240 by DNS cache." job=mfm-monitor-haproxy pid=32983 >>> time="2018-02-02T15:45:02+03:00" level=info msg="[WARNING] 032/154502 >>> (32983) : tsdb_backend_query/tsdb_query3 changed its IP from 10.182.161.240 >>> to 10.182.161.92 by DNS cache." job=mfm-monitor-haproxy pid=32983 >>> time="2018-02-02T15:45:04+03:00" level=info msg="[WARNING] 032/154504 >>> (32983) : tsdb_backend_query/tsdb_query2 changed its IP from 10.182.161.152 >>> to 10.182.161.240 by DNS cache." job=mfm-monitor-haproxy pid=32983 >>> time="2018-02-02T15:45:06+03:00" level=info msg="[WARNING] 032/154506 >>> (32983) : tsdb_backend_query/tsdb_query1 changed its IP from 10.182.161.239 >>> to 10.182.161.152 by DNS cache." job=mfm-monitor-haproxy pid=32983 >>> time="2018-02-02T15:45:10+03:00" level=info msg="[WARNING] 032/154510 >>> (32983) : tsdb_backend_query/tsdb_query3 changed its IP from 10.182.161.92 >>> to 10.182.161.239 by DNS cache." job=mfm-monitor-haproxy pid=32983 >>> time="2018-02-02T15:45:18+03:00" level=info msg="[WARNING] 032/154518 >>> (32983) : tsdb_backend_query/tsdb_query3 changed its IP from 10.182.161.239 >>> to 10.182.161.92 by DNS cache." job=mfm-monitor-haproxy pid=32983 >>> time="2018-02-02T15:45:20+03:00" level=info msg="[WARNING] 032/154520 >>> (32983) : tsdb_backend_query/tsdb_query2 changed its IP from 10.182.161.240 >>> to 10.182.161.239 by DNS cache." job=mfm-monitor-haproxy pid=32983 >>> >>> This isn’t really break the service, but I think this is not quite >>> normal. >>> >>> Any advise on how to resolve this issue? >>> >>> > -- > Mike Chepaykin > >
