Does "all the time" mean on every scrape, or just intermittently?
I've just tested this on a different type of device, with snmpv2 and snmp_exporter 0.20.0. I don't get any errors, although I do get non-standard metric names, e.g. snmp_pxdlrouternxos_ifHCInOctets (if it were me, I'd just call it ifHCInOctets) But otherwise, without a reproducing test case, I'm afraid I can't help you. What I could suggest is that you do a binary chop. Divide the "walk" section into two; test with the first half and then with the second half. If one half causes a crash and the other doesn't, then repeat the chop on the bad half, until you can identify which MIB is causing the issue. On Monday, 21 March 2022 at 10:08:07 UTC [email protected] wrote: > and here is the generator config: pxdlrouternxos generator config > <https://github.com/sapcc/helm-charts/blob/master/prometheus-exporters/snmp-exporter/generator/pxdlrouternxos-generator.yaml> > > On Monday, 21 March 2022 at 09:47:06 UTC+1 [email protected] wrote: > >> Hi Brian, >> >> thanks a lot for the formatting. The config is here: pxdlrouternxos >> config >> <https://github.com/sapcc/helm-charts/blob/master/prometheus-exporters/snmp-exporter/_snmp-exporter-pxdlrouternxos.yaml> >> . >> ~1750 lines. The smp v3 settings are added during helm run via a configmap. >> >> Regards >> >> Olaf >> >> >> On Friday, 18 March 2022 at 18:36:26 UTC+1 Brian Candler wrote: >> >>> Can you share the content of your "pxdlrouternxos" module? >>> >>> Does "all the time" mean on every scrape? >>> >>> Meanwhile: here is your stack trace reformatted. >>> >>> error getting target 10.14.25.10: recover: runtime error: invalid >>> memory address or nil pointer dereference >>> Stack:goroutine 358625 [running]: >>> github.com/gosnmp/gosnmp.(*GoSNMP).send.func1(0xc000e155c8) >>> /go/pkg/mod/github.com/gosnmp/[email protected]/marshal.go:326 >>> +0xa5 >>> panic(0x9e4880, 0xe89790) >>> /usr/local/go/src/runtime/panic.go:969 +0x1b9 >>> >>> github.com/gosnmp/gosnmp.(*GoSNMP).send(0xc0004f6c60, 0xc0004c42a0, >>> 0xc0000de601, 0x0, 0xb34ee0, 0xc000b609c0) >>> /go/pkg/mod/github.com/gosnmp/[email protected]/marshal.go:372 >>> +0x20e >>> >>> github.com/gosnmp/gosnmp.(*GoSNMP).Get(0xc0004f6c60, 0xc00012e300, 0x6, >>> 0x6, 0x0, 0x0, 0xc000522b10) >>> /go/pkg/mod/github.com/gosnmp/[email protected]/gosnmp.go:363 >>> +0x159 >>> >>> main.ScrapeTarget(0xb40660, 0xc0004b41c0, 0xc00045ca67, 0xe, >>> 0xc000161770, 0xb34f40, 0xc000522840, 0x0, 0x0, 0x0, ...) >>> /app/collector.go:133 +0x5be >>> >>> main.collector.Collect(0xb40660, 0xc0004b41c0, 0xc00045ca67, 0xe, >>> 0xc000161770, 0xb34f40, 0xc000522840, 0xc000281020) >>> /app/collector.go:222 +0xbc >>> github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1() >>> /go/pkg/mod/ >>> github.com/prometheus/[email protected]/prometheus/registry.go:446 +0x1a2 >>> created by >>> github.com/prometheus/client_golang/prometheus.(*Registry).Gather >>> /go/pkg/mod/ >>> github.com/prometheus/[email protected]/prometheus/registry.go:457 +0x5ce >>> >>> >>> goroutine 1 [IO wait, 216 minutes]: >>> internal/poll.runtime_pollWait(0x7f0f18245708, 0x72, 0x0) >>> /usr/local/go/src/runtime/netpoll.go:222 +0x55 >>> internal/poll.(*pollDesc).wait(0xc0004c6018, 0x72, 0x0, 0x0, 0xa85b68) >>> /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45 >>> internal/poll.(*pollDesc).waitRead(...) >>> /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 >>> >>> internal/poll.(*FD).Accept(0xc0004c6000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, >>> 0x0) >>> /usr/local/go/src/internal/poll/fd_unix.go:394 +0x1fc >>> net.(*netFD).accept(0xc0004c6000, 0x1eadb908595c5b95, 0x0, 0x0) >>> /usr/local/go/src/net/fd_unix.go:172 +0x45 >>> net.(*TCPListener).accept(0xc0004a8300, 0x6234102b, 0xc00052dad0, >>> 0x48e3a6) >>> /usr/local/go/src/net/tcpsock_posix.go:139 +0x32 >>> net.(*TCPListener).Accept(0xc0004a8300, 0xc00052db20, 0x18, >>> 0xc000000180, 0x7f8dec) >>> /usr/local/go/src/net/tcpsock.go:261 +0x65 >>> net/http.(*Server).Serve(0xc0004c4000, 0xb3ee20, 0xc0004a8300, 0x0, 0x0) >>> /usr/local/go/src/net/http/server.go:2937 +0x266 >>> >>> github.com/prometheus/exporter-toolkit/web.Serve(0xb3ee20, >>> 0xc0004a8300, 0xc0004c4000, 0x0, 0x0, 0xb34f40, 0xc000286420, 0x0, 0xe0) >>> /go/pkg/mod/ >>> github.com/prometheus/[email protected]/web/tls_config.go:192 +0x1b0 >>> >>> github.com/prometheus/exporter-toolkit/web.ListenAndServe(0xc0004c4000, >>> 0x0, 0x0, 0xb34f40, 0xc000286420, 0x0, 0x0) >>> /go/pkg/mod/ >>> github.com/prometheus/[email protected]/web/tls_config.go:184 +0xfd >>> main.main() >>> /app/main.go:248 +0xb71 >>> >>> goroutine 45 [syscall, 3871 minutes]: >>> os/signal.signal_recv(0x0) >>> /usr/local/go/src/runtime/sigqueue.go:147 +0x9d >>> os/signal.loop() >>> /usr/local/go/src/os/signal/signal_unix.go:23 +0x25 >>> created by os/signal.Notify.func1.1 >>> /usr/local/go/src/os/signal/signal.go:150 +0x45 >>> >>> goroutine 46 [select, 3871 minutes]: >>> main.main.func1(0xc00012e360, 0xb34f40, 0xc000286420) >>> /app/main.go:179 +0xe5 >>> created by main.main >>> /app/main.go:177 +0x7ce >>> >>> goroutine 50 [IO wait]: >>> internal/poll.runtime_pollWait(0x7f0f18245620, 0x72, 0xb36220) >>> /usr/local/go/src/runtime/netpoll.go:222 +0x55 >>> internal/poll.(*pollDesc).wait(0xc0004c6098, 0x72, 0xb36200, 0xe444f0, >>> 0x0) >>> /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45 >>> internal/poll.(*pollDesc).waitRead(...) >>> /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 >>> >>> internal/poll.(*FD).Read(0xc0004c6080, 0xc0004d8000, 0x1000, 0x1000, >>> 0x0, 0x0, 0x0) >>> /usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5 >>> >>> net.(*netFD).Read(0xc0004c6080, 0xc0004d8000, 0x1000, 0x1000, 0x92a41b, >>> 0xc00052f7f8, 0x7f2b36) >>> /usr/local/go/src/net/fd_posix.go:55 +0x4f >>> net.(*conn).Read(0xc0004ae050, 0xc0004d8000, 0x1000, 0x1000, 0x0, 0x0, >>> 0x0) >>> /usr/local/go/src/net/net.go:182 +0x8e >>> >>> net/http.(*connReader).Read(0xc0004ac750, 0xc0004d8000, 0x1000, 0x1000, >>> 0x10012864e, 0x0, 0x7f0f18281db8) >>> /usr/local/go/src/net/http/server.go:798 +0x1ad >>> bufio.(*Reader).fill(0xc0002806c0) >>> /usr/local/go/src/bufio/bufio.go:101 +0x105 >>> >>> bufio.(*Reader).ReadSlice(0xc0002806c0, 0xc00025460a, 0x7f0f18281db8, >>> 0xc00052f988, 0x40d950, 0xc00022a100, 0x100) >>> /usr/local/go/src/bufio/bufio.go:360 +0x3d >>> >>> bufio.(*Reader).ReadLine(0xc0002806c0, 0xc00022a100, 0x479294, 0xea09c0, >>> 0x0, 0xa5b4c0, 0xc000730a80) >>> /usr/local/go/src/bufio/bufio.go:389 +0x34 >>> >>> net/textproto.(*Reader).readLineSlice(0xc000730a80, 0xc00022a100, >>> 0x4d788d, 0xc0004c6080, 0x467500, 0xc001bc3e00) >>> /usr/local/go/src/net/textproto/reader.go:58 +0x6c >>> net/textproto.(*Reader).ReadLine(...) >>> /usr/local/go/src/net/textproto/reader.go:39 >>> net/http.readRequest(0xc0002806c0, 0x0, 0xc00022a100, 0x0, 0x0) >>> /usr/local/go/src/net/http/request.go:1012 +0xaa >>> >>> net/http.(*conn).readRequest(0xc00024c0a0, 0xb40660, 0xc0004b4180, 0x0, >>> 0x0, 0x0) >>> /usr/local/go/src/net/http/server.go:984 +0x19a >>> net/http.(*conn).serve(0xc00024c0a0, 0xb40660, 0xc0004b4180) >>> /usr/local/go/src/net/http/server.go:1851 +0x705 >>> created by net/http.(*Server).Serve >>> /usr/local/go/src/net/http/server.go:2969 +0x36c >>> >>> goroutine 358626 [semacquire]: >>> sync.runtime_Semacquire(0xc000880838) >>> /usr/local/go/src/runtime/sema.go:56 +0x45 >>> sync.(*WaitGroup).Wait(0xc000880830) >>> /usr/local/go/src/sync/waitgroup.go:130 +0x65 >>> >>> github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func2(0xc000880830, >>> >>> 0xc000281020, 0xc000281080) >>> /go/pkg/mod/ >>> github.com/prometheus/[email protected]/prometheus/registry.go:463 +0x2b >>> created by >>> github.com/prometheus/client_golang/prometheus.(*Registry).Gather >>> /go/pkg/mod/ >>> github.com/prometheus/[email protected]/prometheus/registry.go:462 +0x60d >>> >>> goroutine 201493 [select]: >>> >>> github.com/prometheus/client_golang/prometheus.(*Registry).Gather(0xc00041a2d0, >>> >>> 0x0, 0x0, 0x0, 0x0, 0x0) >>> /go/pkg/mod/ >>> github.com/prometheus/[email protected]/prometheus/registry.go:513 +0xbbc >>> >>> github.com/prometheus/client_golang/prometheus/promhttp.HandlerFor.func1(0xb3f0a0, >>> >>> 0xc0004c41c0, 0xc00022a000) >>> /go/pkg/mod/ >>> github.com/prometheus/[email protected]/prometheus/promhttp/http.go:126 >>> +0x99 >>> net/http.HandlerFunc.ServeHTTP(0xc0004edc70, 0xb3f0a0, 0xc0004c41c0, >>> 0xc00022a000) >>> /usr/local/go/src/net/http/server.go:2042 +0x44 >>> main.handler(0xb3f0a0, 0xc0004c41c0, 0xc00022a000, 0xb34f40, >>> 0xc000286420) >>> /app/main.go:110 +0x6d7 >>> main.main.func2(0xb3f0a0, 0xc0004c41c0, 0xc00022a000) >>> /app/main.go:201 +0x51 >>> net/http.HandlerFunc.ServeHTTP(0xc0004a82a0, 0xb3f0a0, 0xc0004c41c0, >>> 0xc00022a000) >>> /usr/local/go/src/net/http/server.go:2042 +0x44 >>> net/http.(*ServeMux).ServeHTTP(0xea0040, 0xb3f0a0, 0xc0004c41c0, >>> 0xc00022a000) >>> /usr/local/go/src/net/http/server.go:2417 +0x1ad >>> net/http.serverHandler.ServeHTTP(0xc0004c4000, 0xb3f0a0, 0xc0004c41c0, >>> 0xc00022a000) >>> /usr/local/go/src/net/http/server.go:2843 +0xa3 >>> net/http.(*conn).serve(0xc00024c640, 0xb40660, 0xc0000ca100) >>> /usr/local/go/src/net/http/server.go:1925 +0x8ad >>> created by net/http.(*Server).Serve >>> /usr/local/go/src/net/http/server.go:2969 +0x36c >>> >>> goroutine 358543 [IO wait]: >>> internal/poll.runtime_pollWait(0x7f0f18245368, 0x72, 0xb36220) >>> /usr/local/go/src/runtime/netpoll.go:222 +0x55 >>> internal/poll.(*pollDesc).wait(0xc000290298, 0x72, 0xb36200, 0xe444f0, >>> 0x0) >>> /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45 >>> internal/poll.(*pollDesc).waitRead(...) >>> /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 >>> >>> internal/poll.(*FD).Read(0xc000290280, 0xc000ecf5a1, 0x1, 0x1, 0x0, 0x0, >>> 0x0) >>> /usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5 >>> >>> net.(*netFD).Read(0xc000290280, 0xc000ecf5a1, 0x1, 0x1, 0xc0000ca1d8, >>> 0xc000856f68, 0x4774bc) >>> /usr/local/go/src/net/fd_posix.go:55 +0x4f >>> net.(*conn).Read(0xc00000e040, 0xc000ecf5a1, 0x1, 0x1, 0x0, 0x0, 0x0) >>> /usr/local/go/src/net/net.go:182 +0x8e >>> net/http.(*connReader).backgroundRead(0xc000ecf590) >>> /usr/local/go/src/net/http/server.go:690 +0x58 >>> created by net/http.(*connReader).startBackgroundRead >>> /usr/local/go/src/net/http/server.go:686 +0xd5 >>> >>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/538a3ce9-f480-409f-8edb-ed5b94b804fcn%40googlegroups.com.
