One thought, maybe there's an open file descriptor limit on the snmp_exporter that's being hit?
On Thu, Aug 13, 2020 at 5:18 PM Ben Kochie <[email protected]> wrote: > It looks like the error is coming from the SNMP implementation, when > trying to use a standard net Dialer. > > > https://github.com/soniah/gosnmp/blob/cce23a3123384acf2aebdee032c5ef38698fe290/gosnmp.go#L255-L258 > > If my understanding of the code is correct, this should be using a normal > Golang DNS resolver. > > https://golang.org/pkg/net/#hdr-Name_Resolution > > But the root error "connect: resource temporarily unavailable" is a bit > weird to me. > > On Thu, Aug 13, 2020 at 4:36 PM [email protected] <[email protected]> > wrote: > >> boilerplate.. >> >> - job_name: 'protocols' >> file_sd_configs: >> - files: >> - protocols_targets.json >> metrics_path: /snmp >> params: >> module: [protocols] >> relabel_configs: >> - source_labels: [__address__] >> target_label: __param_target >> - source_labels: [__param_target] >> target_label: instance >> - target_label: __address__ >> replacement: 127.0.0.1:30964 # The SNMP exporter's real hostname:port. >> >> On Thursday, August 13, 2020 at 10:28:11 AM UTC-4 [email protected] wrote: >> >>> Yes, that looks like a DNS issue to me. >>> >>> What is the prometheus.yml configuration for this? >>> >>> On Thu, Aug 13, 2020 at 3:08 PM [email protected] <[email protected]> >>> wrote: >>> >>>> Hello, >>>> >>>> I have snmp exporter with various network devices, which works fine >>>> most of the time. However, sometime I notice false alerts and when I look >>>> at the job, there are gaps of. which snmp exporter doesn't register this as >>>> 0, its just a gap. When looking at snmp exporter logs I noticed these log >>>> messages. >>>> >>>> 2020-08-13_11:06:01.15743 level=info ts=2020-08-13T11:06:01.157Z >>>> caller=collector.go:224 module=protocols target=tor1.mydomain msg="Error >>>> scraping target" err="error connecting to target tor1.mydomain: error >>>> establishing connection to host: dial udp: lookup tor1.mydomain on >>>> 10.1.1.5:53: dial udp 10.1.1.5:53: connect: resource temporarily >>>> unavailable" >>>> >>>> This seems like a DNS issue? Just want to confirm this is in fact the >>>> issue as searching online for this error didn't turn up much. Also if I >>>> have 3 dns servers in my resolve.conf why wouldn't it try each one before >>>> failing? >>>> >>>> Thanks >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Prometheus Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/prometheus-users/79172931-2ca0-48e2-b6ce-bde2800f8896n%40googlegroups.com >>>> <https://groups.google.com/d/msgid/prometheus-users/79172931-2ca0-48e2-b6ce-bde2800f8896n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/3a64e6ee-3b44-4a4f-b986-eb72f73b6d5bn%40googlegroups.com >> <https://groups.google.com/d/msgid/prometheus-users/3a64e6ee-3b44-4a4f-b986-eb72f73b6d5bn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmr4nhXGzOdEAd7fH0e8xHxFj2xRSmDZ-wMjqE6PVoaUUw%40mail.gmail.com.
