OK great. Interesting tcpdump output; I've no idea about the "[!scoped PDU]", or why it doesn't show as "GetBulk" or similar.
On Wednesday, 1 September 2021 at 14:43:24 UTC+1 [email protected] wrote: > yes - you are right, building from main solved the exception: > > level=info ts=2021-09-01T13:34:46.251Z caller=main.go:148 msg="Starting > snmp_exporter" version="(version=, branch=, revision=)" > level=info ts=2021-09-01T13:34:46.251Z caller=main.go:149 > build_context="(go=go1.16.6, user=, date=)" > level=info ts=2021-09-01T13:34:46.304Z caller=main.go:244 msg="Listening > on address" address=:9116 > level=info ts=2021-09-01T13:34:46.304Z caller=tls_config.go:191 msg="TLS > is disabled." http2=false > level=debug ts=2021-09-01T13:39:17.422Z caller=main.go:98 module=ipn > target=10.17.125.48 msg="Starting scrape" > level=debug ts=2021-09-01T13:39:17.422Z caller=collector.go:176 module=ipn > target=10.17.125.48 msg="Getting OIDs" oids=2 > level=info ts=2021-09-01T13:39:57.494Z caller=collector.go:273 module=ipn > target=10.17.125.48 msg="Error scraping target" err="error getting target > 10.17.125.48: not in time window" > level=debug ts=2021-09-01T13:39:57.494Z caller=main.go:109 module=ipn > target=10.17.125.48 msg="Finished scrape" duration_seconds=40.071955962 > > Thanks a lot and I will wait for the next release to roll it out. > > Regards > > Olaf > > On Wednesday, 1 September 2021 at 14:41:19 UTC+2 [email protected] wrote: > >> Can you try with the latest code? There may have been some fixes to the >> Go SNMP library that are not released yet. >> >> On Wed, Sep 1, 2021 at 2:30 PM [email protected] <[email protected]> >> wrote: >> >>> Sorry, I mixed it up with the ips. I am not aware of what is happening, >>> Device itself is reachable, but the response from snmpwalk (Timeout: No >>> Response from 10.17.125.48) or snmp_exporter (nil pointer exception). >>> >>> I am using the tcpdump on the underlying Kubernetes nodes, which is: >>> tcpdump --help >>> tcpdump version 4.9.3 >>> libpcap version 1.9.1 (with TPACKET_V3) >>> OpenSSL 1.1.1k 25 Mar 2021 >>> >>> I am dumping with: "tcpdump -i bond2 host 10.17.125.48 -s 65535 -w >>> dumpnode1" and afterwards I am loading the file into Wireshark and decode >>> it the snmp protocol and the EngineID from the dump and mysnmp v3 settings >>> for this device: >>> >>> I get exactly the same results with that ip: >>> >>> 4 89.998427 10.246.204.141 10.17.125.48 SNMP 195 get-request >>> 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0 >>> 5 89.999266 10.17.125.48 10.246.204.141 SNMP 176 report >>> 1.3.6.1.6.3.15.1.1.2.0 >>> 6 89.999342 10.246.204.141 10.17.125.48 SNMP 198 get-request >>> 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0 >>> 7 99.999515 10.246.204.141 10.17.125.48 SNMP 198 get-request >>> 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0 >>> 8 109.999648 10.246.204.141 10.17.125.48 SNMP 198 get-request >>> 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0 >>> 9 119.999752 10.246.204.141 10.17.125.48 SNMP 198 get-request >>> 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0 >>> 10 149.997236 10.246.204.141 10.17.125.48 SNMP 106 get-request >>> 11 149.998367 10.17.125.48 10.246.204.141 SNMP 147 report >>> 1.3.6.1.6.3.15.1.1.4.0 >>> 12 149.998500 10.246.204.141 10.17.125.48 SNMP 195 get-request >>> 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0 >>> 13 149.999344 10.17.125.48 10.246.204.141 SNMP 176 report >>> 1.3.6.1.6.3.15.1.1.2.0 >>> 14 149.999498 10.246.204.141 10.17.125.48 SNMP 198 get-request >>> 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0 >>> 15 159.999630 10.246.204.141 10.17.125.48 SNMP 198 get-request >>> 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0 >>> 16 169.999816 10.246.204.141 10.17.125.48 SNMP 198 get-request >>> 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0 >>> 17 179.999849 10.246.204.141 10.17.125.48 SNMP 198 get-request >>> 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0 >>> 18 329.997044 10.246.204.141 10.17.125.48 SNMP 106 get-request >>> 19 329.998176 10.17.125.48 10.246.204.141 SNMP 147 report >>> 1.3.6.1.6.3.15.1.1.4.0 >>> 20 329.998309 10.246.204.141 10.17.125.48 SNMP 195 get-request >>> 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0 >>> 21 329.999266 10.17.125.48 10.246.204.141 SNMP 176 report >>> 1.3.6.1.6.3.15.1.1.2.0 >>> 22 329.999339 10.246.204.141 10.17.125.48 SNMP 198 get-request >>> 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0 >>> 23 339.999467 10.246.204.141 10.17.125.48 SNMP 198 get-request >>> 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0 >>> 24 349.999865 10.246.204.141 10.17.125.48 SNMP 198 get-request >>> 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0 >>> 25 359.999957 10.246.204.141 10.17.125.48 SNMP 198 get-request >>> 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0 >>> 26 449.996962 10.246.204.141 10.17.125.48 SNMP 106 get-request >>> 27 449.998087 10.17.125.48 10.246.204.141 SNMP 147 report >>> 1.3.6.1.6.3.15.1.1.4.0 >>> >>> >>> Using your tcpdump command and dumping on the node, where the request >>> comes from: >>> >>> 12:27:57.634370 IP 10.246.204.141.42672 > 10.17.125.48.161: F=r U="" E= >>> C="" GetRequest(14) >>> 0x0000: 4500 005c 7940 4000 3f11 638c 0af6 cc8d E..\y@@.?.c..... >>> 0x0010: 0a11 7d30 a6b0 00a1 0048 5f1e 303e 0201 ..}0.....H_.0>.. >>> 0x0020: 0330 1102 043e ae50 5402 0300 ffff 0401 .0...>.PT....... >>> 0x0030: 0402 0103 0410 300e 0400 0201 0002 0100 ......0......... >>> 0x0040: 0400 0400 0400 3014 0400 0400 a00e 0204 ......0......... >>> 0x0050: 3eae 5054 0201 0002 0100 3000 >.PT......0. >>> 12:27:57.635510 IP 10.17.125.48.161 > 10.246.204.141.42672: F= U="" >>> E=_80_00_00_09_03_00_a3_8e_8c_3e_06 C="" Report(33) >>> .1.3.6.1.6.3.15.1.1.4.0=425050 >>> 0x0000: 4500 0085 d547 0000 3811 4e5c 0a11 7d30 E....G..8.N\..}0 >>> 0x0010: 0af6 cc8d 00a1 a6b0 0071 1fd9 3067 0201 .........q..0g.. >>> 0x0020: 0330 1102 043e ae50 5402 0300 ffe3 0401 .0...>.PT....... >>> 0x0030: 0002 0103 041b 3019 040b 8000 0009 0300 ......0......... >>> 0x0040: a38e 8c3e 0602 0100 0201 0004 0004 0004 ...>............ >>> 0x0050: 0030 3204 0b80 0000 0903 00a3 8e8c 3e06 .02...........>. >>> 0x0060: 0400 a821 0204 3eae 5054 0201 0002 0100 ...!..>.PT...... >>> 0x0070: 3013 3011 060a 2b06 0106 030f 0101 0400 0.0...+......... >>> 0x0080: 4103 067c 5a A..|Z >>> 12:27:57.635651 IP 10.246.204.141.42672 > 10.17.125.48.161: F=apr >>> U="cc-snmpcollector" [!scoped >>> PDU]7f_90_40_01_da_68_03_d3_41_8c_8c_0f_6d_33_73_6d_73_4d_bf_67_b8_bf_0b_2c_ad_83_4f_b0_05_03_0a_fb_cc_2c_cb_b9_33_c0_2f_55_ac_41_b1_a5_ee_45_d6_11_9b_6f_ea_ac_d0_0b_85_5f_ac_fd_9c_52_07 >>> 0x0000: 4500 00b5 7941 4000 3f11 6332 0af6 cc8d E...yA@.?.c2.... >>> 0x0010: 0a11 7d30 a6b0 00a1 00a1 5f77 3081 9602 ..}0......_w0... >>> 0x0020: 0103 3011 0204 3eae 5055 0203 00ff ff04 ..0...>.PU...... >>> 0x0030: 0107 0201 0304 3f30 3d04 0b80 0000 0903 ......?0=....... >>> 0x0040: 00a3 8e8c 3e06 0201 0002 0100 0410 6363 ....>.........cc >>> 0x0050: 2d73 6e6d 7063 6f6c 6c65 6374 6f72 040c -snmpcollector.. >>> 0x0060: 5607 e59f adf6 957c c840 1b52 0408 2838 V......|[email protected]..(8 >>> 0x0070: 8d8a 7de2 066b 043d 7f90 4001 da68 03d3 ..}[email protected].. >>> 0x0080: 418c 8c0f 6d33 736d 734d bf67 b8bf 0b2c A...m3smsM.g..., >>> 0x0090: ad83 4fb0 0503 0afb cc2c cbb9 33c0 2f55 ..O......,..3./U >>> 0x00a0: ac41 b1a5 ee45 d611 9b6f eaac d00b 855f .A...E...o....._ >>> 0x00b0: acfd 9c52 07 ...R. >>> 12:27:57.636499 IP 10.17.125.48.161 > 10.246.204.141.42672: F=a >>> U="cc-snmpcollector" E=_80_00_00_09_03_00_a3_8e_8c_3e_06 C="" Report(30) >>> .1.3.6.1.6.3.15.1.1.2.0=424432 >>> 0x0000: 4500 00a2 d548 0000 3811 4e3e 0a11 7d30 E....H..8.N>..}0 >>> 0x0010: 0af6 cc8d 00a1 a6b0 008e 3f02 3081 8302 ..........?.0... >>> 0x0020: 0103 3011 0204 3eae 5055 0203 00ff e304 ..0...>.PU...... >>> 0x0030: 0101 0201 0304 3a30 3804 0b80 0000 0903 ......:08....... >>> 0x0040: 00a3 8e8c 3e06 0201 0b02 0401 7f09 6e04 ....>.........n. >>> 0x0050: 1063 632d 736e 6d70 636f 6c6c 6563 746f .cc-snmpcollecto >>> 0x0060: 7204 0c6b 68a6 3d05 efde e33c 0dd5 7904 r..kh.=....<..y. >>> 0x0070: 0030 2f04 0b80 0000 0903 00a3 8e8c 3e06 .0/...........>. >>> 0x0080: 0400 a81e 0201 0002 0100 0201 0030 1330 .............0.0 >>> 0x0090: 1106 0a2b 0601 0603 0f01 0102 0041 0306 ...+.........A.. >>> 0x00a0: 79f0 y. >>> 12:27:57.636578 IP 10.246.204.141.42672 > 10.17.125.48.161: F=apr >>> U="cc-snmpcollector" [!scoped >>> PDU]63_dc_d8_88_bc_35_f9_0c_bf_80_e0_80_ce_ef_03_3f_5c_d7_18_93_46_e4_6c_06_49_09_ba_45_73_ca_1e_e2_cc_c8_96_bf_51_7b_3e_7e_65_86_f7_8c_b1_d9_91_fd_4c_72_70_c8_c0_c1_4f_bb_02_13_8e_30_bd >>> 0x0000: 4500 00b8 7942 4000 3f11 632e 0af6 cc8d E...yB@.?.c..... >>> 0x0010: 0a11 7d30 a6b0 00a1 00a4 5f7a 3081 9902 ..}0......_z0... >>> 0x0020: 0103 3011 0204 3eae 5056 0203 00ff ff04 ..0...>.PV...... >>> 0x0030: 0107 0201 0304 4230 4004 0b80 0000 0903 ......B0@....... >>> 0x0040: 00a3 8e8c 3e06 0201 0b02 0401 7f09 6e04 ....>.........n. >>> 0x0050: 1063 632d 736e 6d70 636f 6c6c 6563 746f .cc-snmpcollecto >>> 0x0060: 7204 0ce9 99ca f365 84a6 df1e aa13 9e04 r......e........ >>> 0x0070: 0828 388d 8a7d e206 6c04 3d63 dcd8 88bc .(8..}..l.=c.... >>> 0x0080: 35f9 0cbf 80e0 80ce ef03 3f5c d718 9346 5.........?\...F >>> 0x0090: e46c 0649 09ba 4573 ca1e e2cc c896 bf51 .l.I..Es.......Q >>> 0x00a0: 7b3e 7e65 86f7 8cb1 d991 fd4c 7270 c8c0 {>~e.......Lrp.. >>> 0x00b0: c14f bb02 138e 30bd .O....0. >>> 12:28:07.636786 IP 10.246.204.141.42672 > 10.17.125.48.161: F=apr >>> U="cc-snmpcollector" [!scoped >>> PDU]6e_47_0c_78_2a_e7_10_19_fb_6e_d8_3d_5e_1b_0d_19_89_56_d7_24_46_69_05_31_e1_4c_b8_95_1e_6f_29_08_89_e6_cc_d5_81_42_9a_ce_c8_01_1d_03_65_af_64_de_1d_c1_b6_2e_7b_67_7b_bd_9a_8a_f3_2d_bf >>> 0x0000: 4500 00b8 8e8f 4000 3f11 4de1 0af6 cc8d E.....@.?.M..... >>> 0x0010: 0a11 7d30 a6b0 00a1 00a4 5f7a 3081 9902 ..}0......_z0... >>> 0x0020: 0103 3011 0204 3eae 5057 0203 00ff ff04 ..0...>.PW...... >>> 0x0030: 0107 0201 0304 4230 4004 0b80 0000 0903 ......B0@....... >>> 0x0040: 00a3 8e8c 3e06 0201 0b02 0401 7f09 6e04 ....>.........n. >>> 0x0050: 1063 632d 736e 6d70 636f 6c6c 6563 746f .cc-snmpcollecto >>> 0x0060: 7204 0cd9 42c1 c699 f51d a0db 4d2f bb04 r...B.......M/.. >>> 0x0070: 0828 388d 8a7d e206 6d04 3d6e 470c 782a .(8..}..m.=nG.x* >>> 0x0080: e710 19fb 6ed8 3d5e 1b0d 1989 56d7 2446 ....n.=^....V.$F >>> 0x0090: 6905 31e1 4cb8 951e 6f29 0889 e6cc d581 i.1.L...o)...... >>> 0x00a0: 429a cec8 011d 0365 af64 de1d c1b6 2e7b B......e.d.....{ >>> 0x00b0: 677b bd9a 8af3 2dbf g{....-. >>> 12:28:17.636916 IP 10.246.204.141.42672 > 10.17.125.48.161: F=apr >>> U="cc-snmpcollector" [!scoped >>> PDU]6b_2e_28_d6_6e_16_14_fe_19_2f_7c_8a_50_df_55_6e_c6_ed_75_45_b3_06_32_b7_9f_61_7c_53_52_d0_db_c4_61_3c_f0_42_24_af_63_9b_99_21_40_b3_ac_4c_c2_72_c6_a6_46_6a_84_7c_d2_67_77_70_bc_b4_1a >>> 0x0000: 4500 00b8 a855 4000 3f11 341b 0af6 cc8d E....U@.?.4..... >>> 0x0010: 0a11 7d30 a6b0 00a1 00a4 5f7a 3081 9902 ..}0......_z0... >>> 0x0020: 0103 3011 0204 3eae 5058 0203 00ff ff04 ..0...>.PX...... >>> 0x0030: 0107 0201 0304 4230 4004 0b80 0000 0903 ......B0@....... >>> 0x0040: 00a3 8e8c 3e06 0201 0b02 0401 7f09 6e04 ....>.........n. >>> 0x0050: 1063 632d 736e 6d70 636f 6c6c 6563 746f .cc-snmpcollecto >>> 0x0060: 7204 0cad 6828 91ec bdb9 f9d2 3dae 1704 r...h(......=... >>> 0x0070: 0828 388d 8a7d e206 6e04 3d6b 2e28 d66e .(8..}..n.=k.(.n >>> 0x0080: 1614 fe19 2f7c 8a50 df55 6ec6 ed75 45b3 ..../|.P.Un..uE. >>> 0x0090: 0632 b79f 617c 5352 d0db c461 3cf0 4224 .2..a|SR...a<.B$ >>> 0x00a0: af63 9b99 2140 b3ac 4cc2 72c6 a646 6a84 [email protected]. >>> 0x00b0: 7cd2 6777 70bc b41a |.gwp... >>> 12:28:27.637047 IP 10.246.204.141.42672 > 10.17.125.48.161: F=apr >>> U="cc-snmpcollector" [!scoped >>> PDU]0b_01_37_24_7b_7a_5c_5e_03_5c_e4_ff_79_c5_6f_41_e1_72_06_e8_a0_c0_e9_d5_0c_5b_5a_7e_1c_f7_63_92_f8_3f_92_94_8f_48_0f_6c_69_8c_b8_60_4a_a5_7b_72_ad_9d_32_2e_18_5d_07_42_3e_ec_bd_05_0e >>> 0x0000: 4500 00b8 ac5d 4000 3f11 3013 0af6 cc8d E....]@.?.0..... >>> 0x0010: 0a11 7d30 a6b0 00a1 00a4 5f7a 3081 9902 ..}0......_z0... >>> 0x0020: 0103 3011 0204 3eae 5059 0203 00ff ff04 ..0...>.PY...... >>> 0x0030: 0107 0201 0304 4230 4004 0b80 0000 0903 ......B0@....... >>> 0x0040: 00a3 8e8c 3e06 0201 0b02 0401 7f09 6e04 ....>.........n. >>> 0x0050: 1063 632d 736e 6d70 636f 6c6c 6563 746f .cc-snmpcollecto >>> 0x0060: 7204 0cbf 0ae9 dc09 fbf7 aedc 2c76 2b04 r...........,v+. >>> 0x0070: 0828 388d 8a7d e206 6f04 3d0b 0137 247b .(8..}..o.=..7${ >>> 0x0080: 7a5c 5e03 5ce4 ff79 c56f 41e1 7206 e8a0 z\^.\..y.oA.r... >>> 0x0090: c0e9 d50c 5b5a 7e1c f763 92f8 3f92 948f ....[Z~..c..?... >>> 0x00a0: 480f 6c69 8cb8 604a a57b 72ad 9d32 2e18 H.li..`J.{r..2.. >>> 0x00b0: 5d07 423e ecbd 050e ].B>.... >>> >>> >>> which is the whole communication between the device and the node during >>> the call: >>> >>> level=debug ts=2021-09-01T12:27:57.634Z caller=main.go:102 module=ipn >>> target=10.17.125.48 msg="Starting scrape" >>> level=debug ts=2021-09-01T12:27:57.634Z caller=collector.go:131 >>> module=ipn target=10.17.125.48 msg="Getting OIDs" oids=2 >>> level=info ts=2021-09-01T12:28:37.638Z caller=collector.go:224 >>> module=ipn target=10.17.125.48 msg="Error scraping target" err="error >>> getting target 10.17.125.48: recover: runtime error: invalid memory >>> address or nil pointer dereference\nStack:goroutine 305377 [running]:\ >>> ngithub.com/gosnmp/gosnmp.(*GoSNMP).send.func1(0xc000fc35c8)\n\t/go/pkg/mod/github.com/gosnmp/[email protected]/marshal.go:326 >>> >>> <http://ngithub.com/gosnmp/gosnmp.(*GoSNMP).send.func1(0xc000fc35c8)%5Cn%5Ct/go/pkg/mod/github.com/gosnmp/[email protected]/marshal.go:326> >>> >>> +0xa5\npanic(0x9e4880, 0xe89790)\n\t/usr/local/go/src/runtime/panic.go:969 >>> +0x1b9\ngithub.com/gosnmp/gosnmp.(*GoSNMP).send(0xc00014a240, >>> 0xc0006f2460, 0xc000c21c01, 0x0, 0xb34ee0, 0xc00114a370)\n\t/go/pkg/mod/ >>> github.com/gosnmp/[email protected]/marshal.go:372 >>> <http://github.com/gosnmp/[email protected]/marshal.go:372> +0x20e\ >>> ngithub.com/gosnmp/gosnmp.(*GoSNMP).Get(0xc00014a240, 0xc000bd1c00, >>> 0x2, 0x2, 0x0, 0x0, 0xc000227b60)\n\t/go/pkg/mod/ >>> github.com/gosnmp/[email protected]/gosnmp.go:363 >>> <http://github.com/gosnmp/[email protected]/gosnmp.go:363> >>> +0x159\nmain.ScrapeTarget(0xb40660, 0xc00042c280, 0xc0005c86dc, 0xc, >>> 0xc000189a40, 0xb34f40, 0xc000227890, 0x0, 0x0, 0x0, >>> ...)\n\t/app/collector.go:133 +0x5be\nmain.collector.Collect(0xb40660, >>> 0xc00042c280, 0xc0005c86dc, 0xc, 0xc000189a40, 0xb34f40, 0xc000227890, >>> 0xc0015d69c0)\n\t/app/collector.go:222 +0xbc\ >>> ngithub.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1()\n\t/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:446 >>> >>> <http://ngithub.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1()%5Cn%5Ct/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:446> >>> >>> +0x1a2\ncreated by >>> github.com/prometheus/client_golang/prometheus.(*Registry).Gather\n\t/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:457 >>> >>> <http://github.com/prometheus/client_golang/prometheus.(*Registry).Gather%5Cn%5Ct/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:457> >>> >>> +0x5ce\n\ngoroutine 1 [IO wait, 14 >>> minutes]:\ninternal/poll.runtime_pollWait(0x7f92c58a8eb8, 0x72, >>> 0x0)\n\t/usr/local/go/src/runtime/netpoll.go:222 >>> +0x55\ninternal/poll.(*pollDesc).wait(0xc000c94018, 0x72, 0x0, 0x0, >>> 0xa85b68)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 >>> +0x45\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Accept(0xc000c94000, >>> >>> 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, >>> 0x0)\n\t/usr/local/go/src/internal/poll/fd_unix.go:394 >>> +0x1fc\nnet.(*netFD).accept(0xc000c94000, 0x2deef00cbedce7d5, 0x0, >>> 0x0)\n\t/usr/local/go/src/net/fd_unix.go:172 >>> +0x45\nnet.(*TCPListener).accept(0xc0007fc300, 0x612f6e8c, 0xc00060fad0, >>> 0x48e3a6)\n\t/usr/local/go/src/net/tcpsock_posix.go:139 >>> +0x32\nnet.(*TCPListener).Accept(0xc0007fc300, 0xc00060fb20, 0x18, >>> 0xc000000180, 0x7f8dec)\n\t/usr/local/go/src/net/tcpsock.go:261 >>> +0x65\nnet/http.(*Server).Serve(0xc000c92000, 0xb3ee20, 0xc0007fc300, 0x0, >>> 0x0)\n\t/usr/local/go/src/net/http/server.go:2937 +0x266\ >>> ngithub.com/prometheus/exporter-toolkit/web.Serve(0xb3ee20, >>> 0xc0007fc300, 0xc000c92000, 0x0, 0x0, 0xb34f40, 0xc0000b0c00, 0x0, >>> 0xe0)\n\t/go/pkg/mod/ >>> github.com/prometheus/[email protected]/web/tls_config.go:192 >>> <http://github.com/prometheus/[email protected]/web/tls_config.go:192> >>> >>> +0x1b0\ >>> ngithub.com/prometheus/exporter-toolkit/web.ListenAndServe(0xc000c92000, >>> 0x0, 0x0, 0xb34f40, 0xc0000b0c00, 0x0, 0x0)\n\t/go/pkg/mod/ >>> github.com/prometheus/[email protected]/web/tls_config.go:184 >>> <http://github.com/prometheus/[email protected]/web/tls_config.go:184> >>> >>> +0xfd\nmain.main()\n\t/app/main.go:248 +0xb71\n\ngoroutine 30 [syscall, >>> 1369 >>> minutes]:\nos/signal.signal_recv(0x0)\n\t/usr/local/go/src/runtime/sigqueue.go:147 >>> >>> +0x9d\nos/signal.loop()\n\t/usr/local/go/src/os/signal/signal_unix.go:23 >>> +0x25\ncreated by >>> os/signal.Notify.func1.1\n\t/usr/local/go/src/os/signal/signal.go:150 >>> +0x45\n\ngoroutine 31 [select, 1369 >>> minutes]:\nmain.main.func1(0xc00007c300, 0xb34f40, >>> 0xc0000b0c00)\n\t/app/main.go:179 +0xe5\ncreated by >>> main.main\n\t/app/main.go:177 +0x7ce\n\ngoroutine 153 [select]:\ >>> ngithub.com/prometheus/client_golang/prometheus.(*Registry).Gather(0xc000c210e0, >>> >>> 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/go/pkg/mod/ >>> github.com/prometheus/[email protected]/prometheus/registry.go:513 >>> <http://github.com/prometheus/[email protected]/prometheus/registry.go:513> >>> >>> +0xbbc\ >>> ngithub.com/prometheus/client_golang/prometheus/promhttp.HandlerFor.func1(0xb3f0a0, >>> >>> 0xc000c920e0, 0xc0002ee100)\n\t/go/pkg/mod/ >>> github.com/prometheus/[email protected]/prometheus/promhttp/http.go:126 >>> <http://github.com/prometheus/[email protected]/prometheus/promhttp/http.go:126> >>> >>> +0x99\nnet/http.HandlerFunc.ServeHTTP(0xc0001f6a10, 0xb3f0a0, 0xc000c920e0, >>> 0xc0002ee100)\n\t/usr/local/go/src/net/http/server.go:2042 >>> +0x44\nmain.handler(0xb3f0a0, 0xc000c920e0, 0xc0002ee100, 0xb34f40, >>> 0xc0000b0c00)\n\t/app/main.go:110 +0x6d7\nmain.main.func2(0xb3f0a0, >>> 0xc000c920e0, 0xc0002ee100)\n\t/app/main.go:201 >>> +0x51\nnet/http.HandlerFunc.ServeHTTP(0xc0007fc2a0, 0xb3f0a0, 0xc000c920e0, >>> 0xc0002ee100)\n\t/usr/local/go/src/net/http/server.go:2042 >>> +0x44\nnet/http.(*ServeMux).ServeHTTP(0xea0040, 0xb3f0a0, 0xc000c920e0, >>> 0xc0002ee100)\n\t/usr/local/go/src/net/http/server.go:2417 >>> +0x1ad\nnet/http.serverHandler.ServeHTTP(0xc000c92000, 0xb3f0a0, >>> 0xc000c920e0, 0xc0002ee100)\n\t/usr/local/go/src/net/http/server.go:2843 >>> +0xa3\nnet/http.(*conn).serve(0xc0000df900, 0xb40660, >>> 0xc00028e540)\n\t/usr/local/go/src/net/http/server.go:1925 +0x8ad\ncreated >>> by net/http.(*Server).Serve\n\t/usr/local/go/src/net/http/server.go:2969 >>> +0x36c\n\ngoroutine 38 [select]:\ >>> ngithub.com/prometheus/client_golang/prometheus.(*Registry).Gather(0xc0005703c0, >>> >>> 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/go/pkg/mod/ >>> github.com/prometheus/[email protected]/prometheus/registry.go:513 >>> <http://github.com/prometheus/[email protected]/prometheus/registry.go:513> >>> >>> +0xbbc\ >>> ngithub.com/prometheus/client_golang/prometheus/promhttp.HandlerFor.func1(0xb3f0a0, >>> >>> 0xc0006f2ee0, 0xc009152000)\n\t/go/pkg/mod/ >>> github.com/prometheus/[email protected]/prometheus/promhttp/http.go:126 >>> <http://github.com/prometheus/[email protected]/prometheus/promhttp/http.go:126> >>> >>> +0x99\nnet/http.HandlerFunc.ServeHTTP(0xc0001c61c0, 0xb3f0a0, 0xc0006f2ee0, >>> 0xc009152000)\n\t/usr/local/go/src/net/http/server.go:2042 >>> +0x44\nmain.handler(0xb3f0a0, 0xc0006f2ee0, 0xc009152000, 0xb34f40, >>> 0xc0000b0c00)\n\t/app/main.go:110 +0x6d7\nmain.main.func2(0xb3f0a0, >>> 0xc0006f2ee0, 0xc009152000)\n\t/app/main.go:201 >>> +0x51\nnet/http.HandlerFunc.ServeHTTP(0xc0007fc2a0, 0xb3f0a0, 0xc0006f2ee0, >>> 0xc009152000)\n\t/usr/local/go/src/net/http/server.go:2042 >>> +0x44\nnet/http.(*ServeMux).ServeHTTP(0xea0040, 0xb3f0a0, 0xc0006f2ee0, >>> 0xc009152000)\n\t/usr/local/go/src/net/http/server.go:2417 >>> +0x1ad\nnet/http.serverHandler.ServeHTTP(0xc000c92000, 0xb3f0a0, >>> 0xc0006f2ee0, 0xc009152000)\n\t/usr/local/go/src/net/http/server.go:2843 >>> +0xa3\nnet/http.(*conn).serve(0xc00011e3c0, 0xb40660, >>> 0xc0002ea080)\n\t/usr/local/go/src/net/http/server.go:1925 +0x8ad\ncreated >>> by net/http.(*Server).Serve\n\t/usr/local/go/src/net/http/server.go:2969 >>> +0x36c\n\ngoroutine 305373 [IO >>> wait]:\ninternal/poll.runtime_pollWait(0x7f92c58a82f0, 0x72, >>> 0xb36220)\n\t/usr/local/go/src/runtime/netpoll.go:222 >>> +0x55\ninternal/poll.(*pollDesc).wait(0xc000c94518, 0x72, 0xb36200, >>> 0xe444f0, 0x0)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 >>> +0x45\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc000c94500, >>> >>> 0xc000a8f0f1, 0x1, 0x1, 0x0, 0x0, >>> 0x0)\n\t/usr/local/go/src/internal/poll/fd_unix.go:159 >>> +0x1a5\nnet.(*netFD).Read(0xc000c94500, 0xc000a8f0f1, 0x1, 0x1, >>> 0xc0009b1c18, 0xc000401768, >>> 0x4774bc)\n\t/usr/local/go/src/net/fd_posix.go:55 >>> +0x4f\nnet.(*conn).Read(0xc00024c0e0, 0xc000a8f0f1, 0x1, 0x1, 0x0, 0x0, >>> 0x0)\n\t/usr/local/go/src/net/net.go:182 >>> +0x8e\nnet/http.(*connReader).backgroundRead(0xc000a8f0e0)\n\t/usr/local/go/src/net/http/server.go:690 >>> >>> +0x58\ncreated by >>> net/http.(*connReader).startBackgroundRead\n\t/usr/local/go/src/net/http/server.go:686 >>> >>> +0xd5\n\ngoroutine 305437 [IO >>> wait]:\ninternal/poll.runtime_pollWait(0x7f92c5595c00, 0x72, >>> 0xb36220)\n\t/usr/local/go/src/runtime/netpoll.go:222 >>> +0x55\ninternal/poll.(*pollDesc).wait(0xc000c05618, 0x72, 0xb36200, >>> 0xe444f0, 0x0)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 >>> +0x45\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc000c05600, >>> >>> 0xc000bc2521, 0x1, 0x1, 0x0, 0x0, >>> 0x0)\n\t/usr/local/go/src/internal/poll/fd_unix.go:159 >>> +0x1a5\nnet.(*netFD).Read(0xc000c05600, 0xc000bc2521, 0x1, 0x1, 0xea4f80, >>> 0xea4f90, 0xc000405778)\n\t/usr/local/go/src/net/fd_posix.go:55 >>> +0x4f\nnet.(*conn).Read(0xc000288060, 0xc000bc2521, 0x1, 0x1, 0x0, 0x0, >>> 0x0)\n\t/usr/local/go/src/net/net.go:182 >>> +0x8e\nnet/http.(*connReader).backgroundRead(0xc000bc2510)\n\t/usr/local/go/src/net/http/server.go:690 >>> >>> +0x58\ncreated by >>> net/http.(*connReader).startBackgroundRead\n\t/usr/local/go/src/net/http/server.go:686 >>> >>> +0xd5\n\ngoroutine 57 [select]:\ >>> ngithub.com/prometheus/client_golang/prometheus.(*Registry).Gather(0xc000f40690, >>> >>> 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/go/pkg/mod/ >>> github.com/prometheus/[email protected]/prometheus/registry.go:513 >>> <http://github.com/prometheus/[email protected]/prometheus/registry.go:513> >>> >>> +0xbbc\ >>> ngithub.com/prometheus/client_golang/prometheus/promhttp.HandlerFor.func1(0xb3f0a0, >>> >>> 0xc000c93340, 0xc0067af700)\n\t/go/pkg/mod/ >>> github.com/prometheus/[email protected]/prometheus/promhttp/http.go:126 >>> <http://github.com/prometheus/[email protected]/prometheus/promhttp/http.go:126> >>> >>> +0x99\nnet/http.HandlerFunc.ServeHTTP(0xc000829260, 0xb3f0a0, 0xc000c93340, >>> 0xc0067af700)\n\t/usr/local/go/src/net/http/server.go:2042 +0x4" >>> level=debug ts=2021-09-01T12:28:37.638Z caller=main.go:113 module=ipn >>> target=10.17.125.48 msg="Finished scrape" duration_seconds=40.004547915 >>> On Wednesday, 1 September 2021 at 13:40:03 UTC+2 Brian Candler wrote: >>> >>>> I'm not clear on what you're saying. Originally you said this applied >>>> to devices "which are not reachable". Are you now saying that the device >>>> is reachable and responding to some SNMP queries but not others? Is the >>>> problematic device 10.17.125.14 ? (previously you showed 10.17.125.48) >>>> >>>> I have certainly seen cases where devices can return malformed SNMP >>>> responses in certain situations, e.g. truncated packets, which the SNMP >>>> manager (i.e. client) ignores; it just resends the original request until >>>> it gives up and times out. In fact, I have a device like that on my >>>> network right now. But snmp_exporter isn't giving a panic in that case; >>>> just msg="Error scraping target" err="error walking target x.x.x.x: >>>> request >>>> timeout (after 3 retries)" >>>> >>>> Suggestions: >>>> >>>> * See if you can reproduce the problem with "snmpbulkwalk" (not >>>> "snmpwalk") from net-snmp tools >>>> * See if the problem goes away if you set a lower value for >>>> max_repetitions >>>> <https://github.com/prometheus/snmp_exporter/tree/main/generator#file-format> >>>> >>>> in snmp.yml (the default is 25); the corresponding snmpbulkwalk option is >>>> -Cr <N> and defaults to 10. >>>> >>>> Also, that looks like a very strange version of tcpdump you're using. >>>> Try using the standard Linux tcpdump, on either the docker host or inside >>>> the snmp_exporter container: >>>> tcpdump -i eth0 -nn -s0 -X host 10.17.125.14 >>>> When given the full packet to decode (-s0), tcpdump can sometimes >>>> highlight problems with the ASN formatting, e.g. truncated packets. >>>> >>>> It would also be good to run tcpdump on the target device, or as close >>>> to it as possible, and see if you are getting IP fragmentation or other >>>> oddities. >>>> >>>> I agree that snmp_exporter ideally shouldn't log a panic. Fixing that >>>> would require you to capture the situation which causes it to panic for >>>> you >>>> in a way that's reproducible for a developer. But it's starting to look >>>> like the real problem is that the device itself is giving an invalid SNMP >>>> response, in which case, snmp_exporter wouldn't be able to handle it >>>> anyway; ISTM that catching the panic is a reasonable way to handle the >>>> problem. >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/04d59fc6-bd55-4a2c-b824-bc766dbea6a2n%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/prometheus-users/04d59fc6-bd55-4a2c-b824-bc766dbea6a2n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/8c46a284-b713-4ee9-94d5-9dc7aa431942n%40googlegroups.com.
