I honestly have no idea how Consul works internally. On Wed, Jun 17, 2020 at 11:48 PM Dennis Kelly <[email protected]> wrote:
> I have not yet, that was next on my list. I was more so curious why > consul_exporter would need so many connections to the same three servers > for only 1,000 services (i.e. why not reruse a connection? why don't the > close when done... most are TIME_WAIT). > > > On Tuesday, June 16, 2020 at 11:39:05 PM UTC-7 [email protected] wrote: > >> Have you tried setting the --consul.request-limit to limit the number of >> concurrent connections? >> >> On Wed, Jun 17, 2020 at 6:37 AM Dennis Kelly <[email protected]> >> wrote: >> >>> We have a consul cluster of 3 members and about 1k services. >>> consul_exporter has been using significantly more CPU and is also logging >>> this: >>> >>> level=error ts=2020-06-16T23:56:46.593Z caller=consul_exporter.go:400 >>> msg="Failed to query service health" err="Get >>> \"http://consul.service:8500/v1/health/service/[service >>> name]?stale= >>> <http://consul.service:8500/v1/health/service/kong-portal-awd4235b?stale=>\": >>> context deadline exceeded (Client.Timeout exceeded while awaiting headers)" >>> >>> It is running as a docker container in Nomad. I bumped the CPU resource >>> from the default to 900 MHz and also the consul.timeout to 2s. This has >>> improved things, but we still sporadically receive this error. I haven't >>> had a chance to dig through the entire source yet, but wondering why too >>> consult_exporter has so many open connections to the same 3 consul servers: >>> >>> $ netstat | grep :8500 | wc -l >>> >>> 13653 >>> >>> Why would the connections remain, and also if they do remain, not >>> reused? I suspect we may be hitting up against this issue, but hoping for >>> further clarification: >>> >>> https://github.com/prometheus/consul_exporter/issues/102 >>> >>> Thanks! >>> >>> Dennis >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/ece427fb-99ea-4deb-a99c-60707f2c807dn%40googlegroups.com >>> <https://groups.google.com/d/msgid/prometheus-users/ece427fb-99ea-4deb-a99c-60707f2c807dn%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/4927e749-b9d9-4c97-a2fe-d78fca3810fen%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/4927e749-b9d9-4c97-a2fe-d78fca3810fen%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmrrXeUCTjUnRG7zKo9SEQAjDsuXRbGARh%3DxRccwsOocnQ%40mail.gmail.com.
