Im on Consul 1.0.2. Why do you think this issue is about serving SRV over UDP, rather than about different order of SRV or A records returned by Consul DNS with consecutive requests?
On 11 February 2018 at 18:46, Baptiste <[email protected]> wrote: > Hi, > > What consul version are you using? > I'm facing the same issue in my consul lab. That said, it seems to be a > bug in consul, not able to serve too many SRV records over UDP. > I even triggered a consul crash (using 1.0.5 version). > I'm still investigating this issue and will come back to you as soon as I > have more reliable information. > > Note: please ensure the number of server created by server-template > directive (5 in your case) is above the expected number of server available > in your service. > > Baptiste > > > > On Thu, Feb 8, 2018 at 12:32 AM, Чепайкин Михаил <[email protected]> > wrote: > >> Hi >> >> I’ve changed configuration as you suggested: >> >> backend tsdb_backend_query >> server-template tsdb_query 5 >> _mfm-monitor-opentsdb._tcp.service.mfmconsul:4242 check resolvers dns inter >> 1000 >> >> Logs are kinda different - backend servers now go UP and DOWN, but seems >> the same - ip addresses changing in the same way: >> >> time="2018-02-08T02:12:53+03:00" level=info msg="[WARNING] 038/021253 >> (18208) : Server tsdb_backend_query/tsdb_query1 is going DOWN for >> maintenance (No IP for server ). 2 active and 0 backup servers left. 0 >> sessions active, 0 requeued, 0 remaining in queue." job=mfm-monitor-haproxy >> pid=18208 >> time="2018-02-08T02:12:53+03:00" level=info msg="[WARNING] 038/021253 >> (18208) : tsdb_backend_query/tsdb_query1 changed its IP from 10.182.161.223 >> to 10.182.161.211 by DNS cache." job=mfm-monitor-haproxy pid=18208 >> time="2018-02-08T02:12:53+03:00" level=info msg="[WARNING] 038/021253 >> (18208) : Server tsdb_backend_query/tsdb_query1 administratively READY >> thanks to valid DNS answer." job=mfm-monitor-haproxy pid=18208 >> time="2018-02-08T02:12:53+03:00" level=info msg="[WARNING] 038/021253 >> (18208) : Server tsdb_backend_query/tsdb_query1 >> ('0ab6a1d3.addr.dc1.mfmconsul') is UP/READY (resolves again)." >> job=mfm-monitor-haproxy pid=18208 >> time="2018-02-08T02:12:55+03:00" level=info msg="[WARNING] 038/021255 >> (18208) : Server tsdb_backend_query/tsdb_query3 is going DOWN for >> maintenance (No IP for server ). 2 active and 0 backup servers left. 0 >> sessions active, 0 requeued, 0 remaining in queue." job=mfm-monitor-haproxy >> pid=18208 >> time="2018-02-08T02:12:55+03:00" level=info msg="[WARNING] 038/021255 >> (18208) : tsdb_backend_query/tsdb_query3 changed its IP from 10.182.161.98 >> to 10.182.161.223 by DNS cache." job=mfm-monitor-haproxy pid=18208 >> time="2018-02-08T02:12:55+03:00" level=info msg="[WARNING] 038/021255 >> (18208) : Server tsdb_backend_query/tsdb_query3 administratively READY >> thanks to valid DNS answer." job=mfm-monitor-haproxy pid=18208 >> time="2018-02-08T02:12:55+03:00" level=info msg="[WARNING] 038/021255 >> (18208) : Server tsdb_backend_query/tsdb_query3 >> ('0ab6a1df.addr.dc1.mfmconsul') is UP/READY (resolves again)." >> job=mfm-monitor-haproxy pid=18208 >> time="2018-02-08T02:12:57+03:00" level=info msg="[WARNING] 038/021257 >> (18208) : Server tsdb_backend_query/tsdb_query3 is going DOWN for >> maintenance (No IP for server ). 2 active and 0 backup servers left. 0 >> sessions active, 0 requeued, 0 remaining in queue." job=mfm-monitor-haproxy >> pid=18208 >> time="2018-02-08T02:12:57+03:00" level=info msg="[WARNING] 038/021257 >> (18208) : tsdb_backend_query/tsdb_query3 changed its IP from 10.182.161.223 >> to 10.182.161.98 by DNS cache." job=mfm-monitor-haproxy pid=18208 >> time="2018-02-08T02:12:57+03:00" level=info msg="[WARNING] 038/021257 >> (18208) : Server tsdb_backend_query/tsdb_query3 administratively READY >> thanks to valid DNS answer." job=mfm-monitor-haproxy pid=18208 >> time="2018-02-08T02:12:57+03:00" level=info msg="[WARNING] 038/021257 >> (18208) : Server tsdb_backend_query/tsdb_query3 >> ('0ab6a162.addr.dc1.mfmconsul') is UP/READY (resolves again)." >> job=mfm-monitor-haproxy pid=18208 >> time="2018-02-08T02:13:01+03:00" level=info msg="[WARNING] 038/021301 >> (18208) : Server tsdb_backend_query/tsdb_query1 is going DOWN for >> maintenance (No IP for server ). 2 active and 0 backup servers left. 0 >> sessions active, 0 requeued, 0 remaining in queue." job=mfm-monitor-haproxy >> pid=18208 >> time="2018-02-08T02:13:01+03:00" level=info msg="[WARNING] 038/021301 >> (18208) : tsdb_backend_query/tsdb_query1 changed its IP from 10.182.161.211 >> to 10.182.161.223 by DNS cache." job=mfm-monitor-haproxy pid=18208 >> time="2018-02-08T02:13:01+03:00" level=info msg="[WARNING] 038/021301 >> (18208) : Server tsdb_backend_query/tsdb_query1 administratively READY >> thanks to valid DNS answer." job=mfm-monitor-haproxy pid=18208 >> time="2018-02-08T02:13:01+03:00" level=info msg="[WARNING] 038/021301 >> (18208) : Server tsdb_backend_query/tsdb_query1 >> ('0ab6a1df.addr.dc1.mfmconsul') is UP/READY (resolves again)." >> job=mfm-monitor-haproxy pid=18208 >> time="2018-02-08T02:13:05+03:00" level=info msg="[WARNING] 038/021305 >> (18208) : Server tsdb_backend_query/tsdb_query2 is going DOWN for >> maintenance (No IP for server ). 2 active and 0 backup servers left. 0 >> sessions active, 0 requeued, 0 remaining in queue." job=mfm-monitor-haproxy >> pid=18208 >> time="2018-02-08T02:13:05+03:00" level=info msg="[WARNING] 038/021305 >> (18208) : tsdb_backend_query/tsdb_query2 changed its IP from 10.182.161.163 >> to 10.182.161.211 by DNS cache." job=mfm-monitor-haproxy pid=18208 >> >> Any thoughts? >> >> On 8 February 2018 at 01:25, Baptiste <[email protected]> wrote: >> >> Hi >>> >>> You're not using SRV records and that may be the root cause of your >>> issue. >>> Please try something like this: >>> >>> backend tsdb_backend_query >>> server-template tsdb_query 5 >>> _mfm-monitor-opentsdb._tcp.service.mfmconsul:4242 check resolvers dns inter >>> 1000 >>> >>> if "mfm-monitor-opentsdb" is your service name in consul. >>> >>> Baptiste >>> >>> >>> >>> On Wed, Feb 7, 2018 at 2:52 PM, Чепайкин Михаил <[email protected]> >>> wrote: >>> >>>> Hi! >>>> >>>> I have a Consul as service discovery tool and HAProxy as load balancer. >>>> >>>> In Consul registered a service running on a number of servers, and this >>>> service can be scaled by adding and removing nodes and by moving nodes from >>>> one server to another. >>>> >>>> Consul has DNS service which randomizes responses for services like >>>> that: >>>> >>>> [bux] michep@bux:~$ dig +short mfm-monitor-opentsdb.service.mfmconsul >>>> 10.182.161.239 >>>> 10.182.161.152 >>>> 10.182.161.240 >>>> 10.182.161.92 >>>> [bux] michep@bux:~$ dig +short mfm-monitor-opentsdb.service.mfmconsul >>>> 10.182.161.92 >>>> 10.182.161.152 >>>> 10.182.161.240 >>>> 10.182.161.239 >>>> >>>> In HAProxy 1.8.3 im using server-template configuration, like that: >>>> >>>> resolvers dns >>>> nameserver dns1 ${HAPROXY_NAMESERVER} >>>> hold valid 2s >>>> >>>> backend tsdb_backend_query >>>> server-template tsdb_query 5 mfm-monitor-opentsdb.service.mfmconsul:4242 >>>> check resolvers dns inter 1000 >>>> >>>> And in that case I get alot of warinings in haproxy log: >>>> >>>> time="2018-02-02T15:44:32+03:00" level=info msg="[WARNING] 032/154432 >>>> (32983) : tsdb_backend_query/tsdb_query1 changed its IP from >>>> 10.182.161.240 to 10.182.161.239 by DNS cache." job=mfm-monitor-haproxy >>>> pid=32983 >>>> time="2018-02-02T15:44:42+03:00" level=info msg="[WARNING] 032/154442 >>>> (32983) : tsdb_backend_query/tsdb_query1 changed its IP from >>>> 10.182.161.239 to 10.182.161.240 by DNS cache." job=mfm-monitor-haproxy >>>> pid=32983 >>>> time="2018-02-02T15:44:46+03:00" level=info msg="[WARNING] 032/154446 >>>> (32983) : tsdb_backend_query/tsdb_query3 changed its IP from >>>> 10.182.161.152 to 10.182.161.239 by DNS cache." job=mfm-monitor-haproxy >>>> pid=32983 >>>> time="2018-02-02T15:44:50+03:00" level=info msg="[WARNING] 032/154450 >>>> (32983) : tsdb_backend_query/tsdb_query2 changed its IP from 10.182.161.92 >>>> to 10.182.161.152 by DNS cache." job=mfm-monitor-haproxy pid=32983 >>>> time="2018-02-02T15:44:52+03:00" level=info msg="[WARNING] 032/154452 >>>> (32983) : tsdb_backend_query/tsdb_query3 changed its IP from >>>> 10.182.161.239 to 10.182.161.92 by DNS cache." job=mfm-monitor-haproxy >>>> pid=32983 >>>> time="2018-02-02T15:44:56+03:00" level=info msg="[WARNING] 032/154456 >>>> (32983) : tsdb_backend_query/tsdb_query1 changed its IP from >>>> 10.182.161.240 to 10.182.161.239 by DNS cache." job=mfm-monitor-haproxy >>>> pid=32983 >>>> time="2018-02-02T15:45:00+03:00" level=info msg="[WARNING] 032/154500 >>>> (32983) : tsdb_backend_query/tsdb_query3 changed its IP from 10.182.161.92 >>>> to 10.182.161.240 by DNS cache." job=mfm-monitor-haproxy pid=32983 >>>> time="2018-02-02T15:45:02+03:00" level=info msg="[WARNING] 032/154502 >>>> (32983) : tsdb_backend_query/tsdb_query3 changed its IP from >>>> 10.182.161.240 to 10.182.161.92 by DNS cache." job=mfm-monitor-haproxy >>>> pid=32983 >>>> time="2018-02-02T15:45:04+03:00" level=info msg="[WARNING] 032/154504 >>>> (32983) : tsdb_backend_query/tsdb_query2 changed its IP from >>>> 10.182.161.152 to 10.182.161.240 by DNS cache." job=mfm-monitor-haproxy >>>> pid=32983 >>>> time="2018-02-02T15:45:06+03:00" level=info msg="[WARNING] 032/154506 >>>> (32983) : tsdb_backend_query/tsdb_query1 changed its IP from >>>> 10.182.161.239 to 10.182.161.152 by DNS cache." job=mfm-monitor-haproxy >>>> pid=32983 >>>> time="2018-02-02T15:45:10+03:00" level=info msg="[WARNING] 032/154510 >>>> (32983) : tsdb_backend_query/tsdb_query3 changed its IP from 10.182.161.92 >>>> to 10.182.161.239 by DNS cache." job=mfm-monitor-haproxy pid=32983 >>>> time="2018-02-02T15:45:18+03:00" level=info msg="[WARNING] 032/154518 >>>> (32983) : tsdb_backend_query/tsdb_query3 changed its IP from >>>> 10.182.161.239 to 10.182.161.92 by DNS cache." job=mfm-monitor-haproxy >>>> pid=32983 >>>> time="2018-02-02T15:45:20+03:00" level=info msg="[WARNING] 032/154520 >>>> (32983) : tsdb_backend_query/tsdb_query2 changed its IP from >>>> 10.182.161.240 to 10.182.161.239 by DNS cache." job=mfm-monitor-haproxy >>>> pid=32983 >>>> >>>> This isn’t really break the service, but I think this is not quite >>>> normal. >>>> >>>> Any advise on how to resolve this issue? >>>> >>> -- Mike Chepaykin
