Have you tried setting the --consul.request-limit to limit the number of concurrent connections?
On Wed, Jun 17, 2020 at 6:37 AM Dennis Kelly <[email protected]> wrote: > We have a consul cluster of 3 members and about 1k services. > consul_exporter has been using significantly more CPU and is also logging > this: > > level=error ts=2020-06-16T23:56:46.593Z caller=consul_exporter.go:400 > msg="Failed to query service health" err="Get > \"http://consul.service:8500/v1/health/service/[service > name]?stale= > <http://consul.service:8500/v1/health/service/kong-portal-awd4235b?stale=>\": > context deadline exceeded (Client.Timeout exceeded while awaiting headers)" > > It is running as a docker container in Nomad. I bumped the CPU resource > from the default to 900 MHz and also the consul.timeout to 2s. This has > improved things, but we still sporadically receive this error. I haven't > had a chance to dig through the entire source yet, but wondering why too > consult_exporter has so many open connections to the same 3 consul servers: > > $ netstat | grep :8500 | wc -l > > 13653 > > Why would the connections remain, and also if they do remain, not reused? > I suspect we may be hitting up against this issue, but hoping for further > clarification: > > https://github.com/prometheus/consul_exporter/issues/102 > > Thanks! > > Dennis > > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/ece427fb-99ea-4deb-a99c-60707f2c807dn%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/ece427fb-99ea-4deb-a99c-60707f2c807dn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmo3BXKDNcfHvxOAKPA8fv_RC88hQVw1Kfj-2qNOgrnZNg%40mail.gmail.com.
