Is it possible the snmp.yml timeout is too Low? Scrape_co fig hast 1m but logs say 5s. Maybe you should try and Set timeout: 50s in SNMP.yml, too and retries: 0. Or retries: 3 and timeout: 15s.
Ben Kochie schrieb am Dienstag, 16. April 2024 um 11:20:05 UTC+2: > I've got a new packet debugging option that I've been working on: > https://github.com/prometheus/snmp_exporter/pull/1157 > > On Tue, Apr 16, 2024 at 10:56 AM Nicolas <work.nicol...@gmail.com> wrote: > >> Hi again, >> >> So I confirm that the 1.3.6.1.2.1.1 doesn't work fine but I don't know >> why... >> >> *debug.log* : >> >> >> *214 level=debug auth=cisco_v3 target=xx.xx.xx.xx module=arte_mib >> msg="Walking subtree" oid=1.3.6.1.2.1.1393 level=info auth=cisco_v3 target= >> xx.xx.xx.xx module=arte_mib msg="Error scraping target" err="error >> walking target xx.xx.xx.xx: request timeout (after 3 retries)"464 >> level=debug auth=cisco_v3 target= xx.xx.xx.xx module=arte_mib >> msg="Finished scrape" duration_seconds=20.048886702* >> >> *$ snmpwalk -v3 -l authPriv -u user -a SHA -A secret -x AES -X secret >> xx.xx.xx.xx 1.3.6.1.2.1.1* >> >> >> >> >> >> >> >> >> >> >> >> >> >> *SNMPv2-MIB::sysDescr.0 = STRING: Cisco IOS Software [Cupertino], >> Catalyst L3 Switch Software (CAT9K_IOSXE), Version 17.9.4, RELEASE SOFTWARE >> (fc5)Technical Support: http://www.cisco.com/techsupport >> <http://www.cisco.com/techsupport>Copyright (c) 1986-2023 by Cisco Systems, >> Inc.Compiled Wed 26-Jul-23 10:26 by mcpreSNMPv2-MIB::sysObjectID.0 = OID: >> SNMPv2-SMI::enterprises.9.1.2494DISMAN-EVENT-MIB::sysUpTimeInstance = >> Timeticks: (725228220) 83 days, 22:31:22.20SNMPv2-MIB::sysContact.0 = >> STRING:SNMPv2-MIB::sysName.0 = STRING: xx.xx.xx.xxSNMPv2-MIB::sysLocation.0 >> = STRING: SITESNMPv2-MIB::sysServices.0 = INTEGER: >> 6SNMPv2-MIB::sysORLastChange.0 = Timeticks: (0) >> 0:00:00.00SNMPv2-MIB::sysORID.1 = OID: >> SNMPv2-SMI::enterprises.9.7.129SNMPv2-MIB::sysORID.2 = OID: >> SNMPv2-SMI::enterprises.9.7.115SNMPv2-MIB::sysORID.3 = OID: >> SNMPv2-SMI::enterprises.9.7.265* >> >> *generator.yml* >> >> >> >> >> >> >> >> >> >> >> >> >> *auths: cisco_v3: security_level: authPriv username: usesr >> password: secret auth_protocol: SHA priv_protocol: AES >> priv_password: secret version: 3modules: arte_mib: walk: - >> 1.3.6.1.2.1.1* >> >> *./generator generate -m /usr/share/snmp/mibs/ -g generator.yml -o >> snmp.yml* >> >> >> >> *ts=2024-04-16T08:34:31.972Z caller=net_snmp.go:175 level=info >> msg="Loading MIBs" from=/usr/share/snmp/mibs/ts=2024-04-16T08:34:32.016Z >> caller=main.go:53 level=info msg="Generating config for module" >> module=arte_mibts=2024-04-16T08:34:32.018Z caller=main.go:68 level=info >> msg="Generated metrics" module=arte_mib >> metrics=12ts=2024-04-16T08:34:32.019Z caller=main.go:93 level=info >> msg="Config written" >> file=/etc/prometheus/snmp_generator/snmp_exporter-0.25.0/generator/snmp.yml* >> >> *snmp.yml (generated by generator) * >> # WARNING: This file was auto-generated using snmp_exporter generator, >> manual changes will be lost. >> auths: >> cisco_v3: >> community: public >> security_level: authPriv >> username: user >> password: secret >> auth_protocol: SHA >> priv_protocol: AES >> priv_password: secret >> version: 3 >> modules: >> arte_mib: >> walk: >> - 1.3.6.1.2.1.1 >> metrics: >> - name: sysDescr >> oid: 1.3.6.1.2.1.1.1 >> type: DisplayString >> help: A textual description of the entity - 1.3.6.1.2.1.1.1 >> - name: sysObjectID >> oid: 1.3.6.1.2.1.1.2 >> type: OctetString >> help: The vendor's authoritative identification of the network >> management subsystem >> contained in the entity - 1.3.6.1.2.1.1.2 >> - name: sysUpTime >> oid: 1.3.6.1.2.1.1.3 >> type: gauge >> help: The time (in hundredths of a second) since the network >> management portion >> of the system was last re-initialized. - 1.3.6.1.2.1.1.3 >> - name: sysContact >> oid: 1.3.6.1.2.1.1.4 >> type: DisplayString >> help: The textual identification of the contact person for this >> managed node, >> together with information on how to contact this person - >> 1.3.6.1.2.1.1.4 >> - name: sysName >> oid: 1.3.6.1.2.1.1.5 >> type: DisplayString >> help: An administratively-assigned name for this managed node - >> 1.3.6.1.2.1.1.5 >> - name: sysLocation >> oid: 1.3.6.1.2.1.1.6 >> type: DisplayString >> help: The physical location of this node (e.g., 'telephone closet, >> 3rd floor') >> - 1.3.6.1.2.1.1.6 >> - name: sysServices >> oid: 1.3.6.1.2.1.1.7 >> type: gauge >> help: A value which indicates the set of services that this entity >> may potentially >> offer - 1.3.6.1.2.1.1.7 >> - name: sysORLastChange >> oid: 1.3.6.1.2.1.1.8 >> type: gauge >> help: The value of sysUpTime at the time of the most recent change >> in state >> or value of any instance of sysORID. - 1.3.6.1.2.1.1.8 >> - name: sysORIndex >> oid: 1.3.6.1.2.1.1.9.1.1 >> type: gauge >> help: The auxiliary variable used for identifying instances of the >> columnar >> objects in the sysORTable. - 1.3.6.1.2.1.1.9.1.1 >> indexes: >> - labelname: sysORIndex >> type: gauge >> - name: sysORID >> oid: 1.3.6.1.2.1.1.9.1.2 >> type: OctetString >> help: An authoritative identification of a capabilities statement >> with respect >> to various MIB modules supported by the local SNMP application >> acting as a >> command responder. - 1.3.6.1.2.1.1.9.1.2 >> indexes: >> - labelname: sysORIndex >> type: gauge >> - name: sysORDescr >> oid: 1.3.6.1.2.1.1.9.1.3 >> type: DisplayString >> help: A textual description of the capabilities identified by the >> corresponding >> instance of sysORID. - 1.3.6.1.2.1.1.9.1.3 >> indexes: >> - labelname: sysORIndex >> type: gauge >> - name: sysORUpTime >> oid: 1.3.6.1.2.1.1.9.1.4 >> type: gauge >> help: The value of sysUpTime at the time this conceptual row was >> last instantiated. >> - 1.3.6.1.2.1.1.9.1.4 >> indexes: >> - labelname: sysORIndex >> type: gauge >> >> >> >> >> and the tcpdump receive the anwer so why the snmp_exporter says "timeout >> after 3 retries" ? : >> *tcpdump -i ens192 port 161 -l | grep xx.xx.xx.xx* >> dropped privs to tcpdump >> tcpdump: verbose output suppressed, use -v[v]... for full protocol decode >> listening on ens192, link-type EN10MB (Ethernet), snapshot length 262144 >> bytes >> 10:48:02.349225 IP* prometheus01.com.54725 > xx.xx.xx.xx .snmp*: F=r >> U="" E= C="" GetRequest(14) >> 10:48:02.356737 IP *xx.xx.xx.xx.snmp > prometheus01.com.54725*: F= U="" >> E=_80_00_00_09_03_00_68_e5_9e_b2_94_80 C="" Report(33) >> S:snmpUsmMIB.usmMIBObjects.usmStats.usmStatsUnknownEngineIDs.0=1140991 >> 10:48:02.356932 IP *prometheus01.com.54725 > xx.xx.xx.xx .snmp*: >> F=apr U="user" [!scoped >> PDU]a3_f0_93_62_b2_02_47_2b_41_f5_a2_bc_16_97_ef_9f_2b_1c_bd_2e_5c_7b_d8_75_84_0a_de_bd_ff_02_49_dc_7c_2e_2d_3b_13_0f_fb_ea_e5_98_76_5e_b8_fb >> 10:48:02.365440 IP *xx.xx.xx.xx.snmp > prometheus01.com.54725*: F=ap >> U="user" [!scoped >> PDU]0e_b5_6c_00_23_61_81_8c_82_2e_44_75_78_1f_54_5b_22_dc_fe_51_19_2e_1b_40_62_16_41_05_89_f0_d6_43_5d_af_97_25_1e_66_60_6b_bc_61_55_8f_bd_31_76_4b_62_ca_a8_77_26_fc_f8_e8_35_4f_77_9f_b7_25_6e_1d_7f_06_27_07_3e_bc_04_ee_13_d2_d3_43_ef_6f_04_fd_88_03_bb_37_f9_fe_75_9e_0b_c5_99_38_5d_64_af_d1_67_35_3c_99_ff_7b_cc_92_2d_67_10_13_11_3b_f5_85_bd_77_f2_7d_19_67_e8_7d_92_e5_9c_cd_ba_86_eb_d3_23_81_d0_09_e9_be_82_58_6a_e9_b3_c4_1a_78_09_1e_44_bb_f7_fd_23_ed_4c_14_23_f1_83_63_55_aa_b1_c7_ce_7d_b5_94_c9_e4_1a_f3_d9_42_dd_19_24_69_13_ >> Le mardi 16 avril 2024 à 00:24:54 UTC+2, Nicolas a écrit : >> >>> I think i found something. It's the mib of SNMPv2-MIB : 1.3.6.1.2.1.1 >>> which dosen't work fine in the generator. >>> If I'm more precise and give the oid 1.2.6.2.1.1.1 it works for example. >>> The other oids I put next >>> >>> >>> >>> >>> >>> >>> >>> * - 1.3.6.1.2.1.2.2 - 1.3.6.1.2.1.31.1.1 - >>> 1.3.6.1.4.1.9.9.109.1.1.1.1.8 - 1.3.6.1.4.1.9.9.13.1.4 - >>> 1.3.6.1.4.1.9.9.13.1.5 - 1.3.6.1.4.1.9.9.48.1.1.1.5 - >>> 1.3.6.1.4.1.9.9.48.1.1.1.6 - 1.3.6.1.4.1.9.9.68.1.2.2.1.2* >>> work without a hitch. >>> >>> >>> >>> Le lundi 15 avril 2024 à 23:12:18 UTC+2, Nicolas a écrit : >>> >>>> Hi Ben, >>>> Here the debug log, but they are strange too because with an snmwalk >>>> everything is fine. >>>> >>>> *debug log *: >>>> *Apr 15 22:56:20 prometheus01 snmp_exporter[16444]: >>>> ts=2024-04-15T20:56:20.905Z caller=collector.go:393 level=info >>>> auth=cisco_v3 target=* *xx.xx.xx.xx* >>>> * module=arte_mib msg="Error scraping target" err="error walking target >>>> xx.xx.xx.xx: request timeout (after 3 retries)"Apr 15 22:56:20 >>>> prometheus01 >>>> snmp_exporter[16444]: ts=2024-04-15T20:56:20.905Z caller=collector.go:464 >>>> level=debug auth=cisco_v3 target=* *xx.xx.xx.xx* >>>> * module=arte_mib msg="Finished scrape" duration_seconds=20.03805335Apr >>>> 15 22:56:21 prometheus01 snmp_exporter[16444]: ts=2024-04-15T20:56:21.907Z >>>> caller=collector.go:393 level=info auth=cisco_v3 target=* *yy.yy.yy.yy* >>>> >>>> * module=arte_mib msg="Error scraping target" err="error walking target >>>> yy.yy.yy.yy: request timeout (after 3 retries)"Apr 15 22:56:21 >>>> prometheus01 >>>> snmp_exporter[16444]: ts=2024-04-15T20:56:21.907Z caller=collector.go:464 >>>> level=debug auth=cisco_v3 target=* *yy.yy.yy.yy* >>>> >>>> * module=arte_mib msg="Finished scrape" >>>> duration_seconds=20.083473958Apr 15 22:56:22 prometheus01 >>>> snmp_exporter[16444]: ts=2024-04-15T20:56:22.869Z caller=collector.go:460 >>>> level=debug auth=cisco_v3 target=ww.ww.ww.ww module=arte_mib msg="Starting >>>> scrape"Apr 15 22:56:22 prometheus01 snmp_exporter[16444]: >>>> ts=2024-04-15T20:56:22.869Z caller=collector.go:214 level=debug >>>> auth=cisco_v3 target=* *ww.ww.ww.ww* >>>> *module=arte_mib msg="Walking subtree" oid=1.3.6.1.2.1.1Apr 15 22:56:27 >>>> prometheus01 snmp_exporter[16444]: ts=2024-04-15T20:56:27.142Z >>>> caller=collector.go:393 level=info auth=cisco_v3 target=zz.zz.zz.zz >>>> module=arte_mib msg="Error scraping target" err="error walking target * >>>> * zz.zz.zz.zz * >>>> *: request timeout (after 3 retries)"Apr 15 22:56:27 prometheus01 >>>> snmp_exporter[16444]: ts=2024-04-15T20:56:27.142Z caller=collector.go:464 >>>> level=debug auth=cisco_v3 target=* *zz.zz.zz.zz* * module=arte_mib >>>> msg="Finished scrape" duration_seconds=20.075515731* >>>> >>>> *snmpwalk :* >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> *SNMPv2-MIB::sysDescr.0 = STRING: Cisco IOS Software [Cupertino], >>>> Catalyst L3 Switch Software (CAT9K_IOSXE), Version 17.9.4, RELEASE >>>> SOFTWARE >>>> (fc5)Technical Support: http://www.cisco.com/techsupport >>>> <http://www.cisco.com/techsupport>Copyright (c) 1986-2023 by Cisco >>>> Systems, >>>> Inc.Compiled Wed 26-Jul-23 10:26 by mcpreSNMPv2-MIB::sysObjectID.0 = OID: >>>> SNMPv2-SMI::enterprises.9.1.2494DISMAN-EVENT-MIB::sysUpTimeInstance = >>>> Timeticks: (235735378) 27 days, 6:49:13.78SNMPv2-MIB::sysContact.0 = >>>> STRING:SNMPv2-MIB::sysName.0 = STRING: swro-arte-strg-data-1.net.com >>>> <http://swro-arte-strg-data-1.net.globecast.com/>SNMPv2-MIB::sysLocation.0 >>>> = STRING: STRGSNMPv2-MIB::sysServices.0 = INTEGER: >>>> 6SNMPv2-MIB::sysORLastChange.0 = Timeticks: (0) >>>> 0:00:00.00SNMPv2-MIB::sysORID.1 = OID: >>>> SNMPv2-SMI::enterprises.9.7.129SNMPv2-MIB::sysORID.2 = OID: >>>> SNMPv2-SMI::enterprises.9.7.115SNMPv2-MIB::sysORID.3 = OID: >>>> SNMPv2-SMI::enterprises.9.7.265SNMPv2-MIB::sysORID.4 = OID: >>>> SNMPv2-SMI::enterprises.9.7.112SNMPv2-MIB::sysORID.5 = OID: >>>> SNMPv2-SMI::enterprises.9.7.106SNMPv2-MIB::sysORID.6 = OID: >>>> SNMPv2-SMI::enterprises.9.7.582* >>>> ... >>>> >>>> The snmpwalk response is instantaneous, I don't think it's a >>>> scrape_interval and timeout problem. In fact, I've even increased the >>>> interval to 1 minute and it doesn't change a thing. >>>> >>>> I also use tcpdump to see the communication between my equipments and >>>> prometheus and everything seems fine : >>>> *[~]# tcpdump -i ens192 port 161* >>>> >>>> >>>> >>>> *22:43:47.138346 IP prometheus01.com <http://prometheus01.com/>.44170 >>>> > swro-arte-strg-data-2.net.com >>>> <http://swro-arte-strg-data-2.net.com/>.snmp: F=r U="" E= C="" >>>> GetRequest(14)22:43:47.154121 IP swro-arte-strg-data-2.net.com >>>> <http://swro-arte-strg-data-2.net.com/>.snmp > prometheus01.com >>>> <http://prometheus01.com/>.44170: F= U="" >>>> E=_80_00_00_09_03_00_68_e5_9e_a4_00_00 C="" Report(33) >>>> >>>> S:snmpUsmMIB.usmMIBObjects.usmStats.usmStatsUnknownEngineIDs.0=71669222:43:47.204598 >>>> >>>> IP prometheus01.com <http://prometheus01.com/>.44170 >>>> > swro-arte-strg-data-2.net.com >>>> <http://swro-arte-strg-data-2.net.com/>.snmp: F=apr U="supserver" >>>> [!scoped >>>> PDU]9d_5d_4e_0c_1f_26_92_e6_a6_65_0c_3a_04_b0_8e_f7_66_7d_18_97_3f_5d_84_e2_04....* >>>> >>>> >>>> >>>> *......80 packets captured85 packets received by filter0 packets >>>> dropped by kernel* >>>> >>>> Le lundi 15 avril 2024 à 22:41:24 UTC+2, Ben Kochie a écrit : >>>> >>>>> If you use `snmp_exporter --log.level=debug`, what do the logs say? >>>>> >>>>> On Mon, Apr 15, 2024 at 10:38 PM Nicolas <work.nicol...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hello, >>>>>> I have a strange error and I hope you can help me, or maybe there is >>>>>> a problem with the snmp_exporter generator in the latest version. >>>>>> >>>>>> I'm using snmp_exporter version 0.25.0 and prometheus >>>>>> My generator.yml file looks like this: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> *auths: cisco_v3: security_level: authPriv username: >>>>>> secret_name password: secret auth_protocol: SHA priv_protocol: >>>>>> AES priv_password: secret version: 3modules: arte_mib: walk: >>>>>> - >>>>>> 1.3.6.1.2.1.1' - 1.3.6.1.2.1.2.2 - 1.3.6.1.2.1.31.1.1 - >>>>>> 1.3.6.1.4.1.9.9.109.1.1.1.1.8 - 1.3.6.1.4.1.9.9.13.1.4 - >>>>>> 1.3.6.1.4.1.9.9.13.1.5 - 1.3.6.1.4.1.9.9.48.1.1.1.5 - >>>>>> 1.3.6.1.4.1.9.9.48.1.1.1.6 - 1.3.6.1.4.1.9.9.68.1.2.2.1.2* >>>>>> >>>>>> I generate my snmp.yml file without error: >>>>>> >>>>>> >>>>>> >>>>>> *ts=2024-04-15T20:29:38.537Z caller=net_snmp.go:175 level=info >>>>>> msg="Loading MIBs" from=/usr/share/snmp/mibs/ts=2024-04-15T20:29:38.577Z >>>>>> caller=main.go:53 level=info msg="Generating config for module" >>>>>> module=arte_mibts=2024-04-15T20:29:38.580Z caller=main.go:68 level=info >>>>>> msg="Generated metrics" module=arte_mib >>>>>> metrics=64ts=2024-04-15T20:29:38.585Z caller=main.go:93 level=info >>>>>> msg="Config written" >>>>>> file=/etc/prometheus/snmp_generator/snmp_exporter-0.25.0/generator/snmp.yml* >>>>>> >>>>>> And when I push it in my prometheus (conf like this) >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> * - job_name: 'arte_snmp' scrape_interval: 1m scrape_timeout: >>>>>> 50s file_sd_configs: - files: - >>>>>> '/etc/prometheus/targets/arte.json' metrics_path: /snmp params: >>>>>> >>>>>> auth: [cisco_v3] module: [arte_mib] relabel_configs: - >>>>>> source_labels: [__address__] target_label: instance - >>>>>> source_labels: [__address__] target_label: __param_target - >>>>>> source_labels: [module] target_label: __param_module - >>>>>> target_label: __address__ replacement: 127.0.0.1:9116 >>>>>> <http://127.0.0.1:9116> * >>>>>> I get a server returned HTTP status 500 Internal Server Error >>>>>> [image: 500.PNG] >>>>>> >>>>>> >>>>>> Note that if, for example, I use the if_mib module and take the >>>>>> snmp.yml provided in the snmp_exporter, I get no error and my servers >>>>>> appear to be UP. >>>>>> >>>>>> Thanks for your help >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Prometheus Users" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to prometheus-use...@googlegroups.com. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/prometheus-users/711dd213-92de-4abb-911c-883c0e4535edn%40googlegroups.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/prometheus-users/711dd213-92de-4abb-911c-883c0e4535edn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to prometheus-use...@googlegroups.com. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/d7f8ebd2-9205-4d8b-970c-eeb8ff15fccen%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/d7f8ebd2-9205-4d8b-970c-eeb8ff15fccen%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/7ab5486d-4a80-4f93-a708-3fd1ed9340fan%40googlegroups.com.
