I have not yet, that was next on my list. I was more so curious why consul_exporter would need so many connections to the same three servers for only 1,000 services (i.e. why not reruse a connection? why don't the close when done... most are TIME_WAIT).
On Tuesday, June 16, 2020 at 11:39:05 PM UTC-7 [email protected] wrote: > Have you tried setting the --consul.request-limit to limit the number of > concurrent connections? > > On Wed, Jun 17, 2020 at 6:37 AM Dennis Kelly <[email protected]> wrote: > >> We have a consul cluster of 3 members and about 1k services. >> consul_exporter has been using significantly more CPU and is also logging >> this: >> >> level=error ts=2020-06-16T23:56:46.593Z caller=consul_exporter.go:400 >> msg="Failed to query service health" err="Get >> \"http://consul.service:8500/v1/health/service/[service >> name]?stale= >> <http://consul.service:8500/v1/health/service/kong-portal-awd4235b?stale=>\": >> >> context deadline exceeded (Client.Timeout exceeded while awaiting headers)" >> >> It is running as a docker container in Nomad. I bumped the CPU resource >> from the default to 900 MHz and also the consul.timeout to 2s. This has >> improved things, but we still sporadically receive this error. I haven't >> had a chance to dig through the entire source yet, but wondering why too >> consult_exporter has so many open connections to the same 3 consul servers: >> >> $ netstat | grep :8500 | wc -l >> >> 13653 >> >> Why would the connections remain, and also if they do remain, not reused? >> I suspect we may be hitting up against this issue, but hoping for further >> clarification: >> >> https://github.com/prometheus/consul_exporter/issues/102 >> >> Thanks! >> >> Dennis >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/ece427fb-99ea-4deb-a99c-60707f2c807dn%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/ece427fb-99ea-4deb-a99c-60707f2c807dn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/4927e749-b9d9-4c97-a2fe-d78fca3810fen%40googlegroups.com.
