Hi All, I tried commenting it out but it didn't solve the problem that I faced. I'll try 2.37.5 LTS version, if it solves then i'll update you. Thanks
13 Aralık 2022 Salı tarihinde saat 18:42:40 UTC+3 itibarıyla Brian Candler şunları yazdı: > Thank you for the full files (although you probably should have scrubbed > your snmp community) > > The only slightly suspicious thing I see is > honor_timestamps: true > which generally shouldn't be needed except in very special circumstances. > I don't *think* it will make any difference here, since snmp_exporter > doesn't give timestamps with its metrics anyway, but can you try commenting > it out? > > Otherwise, follow Julien's advice of trying the same configuration with > v2.37.5 (which is an LTS release) and see if the problem recurs. > > On Tuesday, 13 December 2022 at 15:28:37 UTC [email protected] wrote: > >> Hi All, >> >> I added my all config files attached. >> Thank you so much for this deep diving. >> >> 13 Aralık 2022 Salı tarihinde saat 17:37:43 UTC+3 itibarıyla Brian >> Candler şunları yazdı: >> >>> I ran your curl output through "promtool check metrics", but it only >>> warns about label and metric naming conventions (which SNMP violates to be >>> consistent with standard MIB usage). There are no duplicate timeseries. >>> >>> # /opt/prometheus/promtool check metrics <ert3.txt 2>&1 | sort | uniq -c >>> 11247 ifAlias label names should be written in 'snake_case' not >>> 'camelCase' >>> 1 ifAlias metric names should be written in 'snake_case' not >>> 'camelCase' >>> 13767 ifDescr label names should be written in 'snake_case' not >>> 'camelCase' >>> 1 ifDescr metric names should be written in 'snake_case' not >>> 'camelCase' >>> 1 ifHCInOctets counter metrics should have "_total" suffix >>> 11247 ifHCInOctets label names should be written in 'snake_case' not >>> 'camelCase' >>> 1 ifHCInOctets metric names should be written in 'snake_case' not >>> 'camelCase' >>> 1 ifHCOutOctets counter metrics should have "_total" suffix >>> 11247 ifHCOutOctets label names should be written in 'snake_case' not >>> 'camelCase' >>> 1 ifHCOutOctets metric names should be written in 'snake_case' not >>> 'camelCase' >>> 1 sysName label names should be written in 'snake_case' not >>> 'camelCase' >>> 1 sysName metric names should be written in 'snake_case' not >>> 'camelCase' >>> 1 sysUpTime metric names should be written in 'snake_case' not >>> 'camelCase' >>> >>> Changing the target.yml file periodically won't make a difference; >>> Prometheus will just pick up the new one for the next scrape. So this is >>> odd. >>> >>> On Tuesday, 13 December 2022 at 12:21:15 UTC [email protected] wrote: >>> >>>> Thanks for your detail explanation. You can see my answers below. >>>> >>>> I am not doing federation or any sort of availability. >>>> I've checked [1h] then [1d] and haven't seen any multiple data points :( >>>> I am not scraping the same targets twice but i am updating target.yml >>>> every hour. Does it affect? >>>> i am not sure there is any bug in my exporter config but it scrapes >>>> data from mydevices every 5 minutes. >>>> Please find curl output attached. >>>> >>>> >>>> 13 Aralık 2022 Salı tarihinde saat 14:44:31 UTC+3 itibarıyla Brian >>>> Candler şunları yazdı: >>>> >>>>> The first error explicitly mentions a particular metric "fix:min_in", >>>>> and the input to the recording rule which generates that is the metric >>>>> "access:ifHCInOctets:FIX_rate" (rule 16, counting from zero) >>>>> But it also mentions an error for the metric >>>>> "access:ifHCInOctets:FIX_rate", which takes its input from "ifHCInOctets" >>>>> (rule 4) >>>>> >>>>> So my guess is that you have duplicate data somewhere, posisbly in >>>>> ifHCInOctets. Try these queries: >>>>> >>>>> >>>>> ifHCInOctets{instance="10.85.4.3",ifAlias="FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||"}[1h] >>>>> >>>>> access:ifHCInOctets:FIX_rate{instance="10.85.4.3",ifAlias="FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||"}[1h] >>>>> >>>>> Do you see multiple data points with the same timestamp? In which >>>>> timeseries? if not, try extending the time range to [1d] >>>>> >>>>> This would however require you to have multiple values of ifHCInOctets >>>>> with the same value of all labels (same ifAlias, same ifDescr, same >>>>> ifIndex). >>>>> >>>>> If that's what you're seeing then you have to work backwards: are you >>>>> scraping the same target twice? Or is there a bug in your snmp_exporter >>>>> config? Try scraping the device manually using curl: >>>>> >>>>> curl -g 'localhost:9115/snmp?module=XXXX&target=10.85.4.3' >>>>> >>>>> (Having said that, if there were duplicates at this point, I'd be >>>>> expecting them to be rejected at scrape time). >>>>> >>>>> Are you doing any sort of high-availability or federation? If so, you >>>>> need different external labels for each data source. >>>>> >>>>> On Tuesday, 13 December 2022 at 10:55:19 UTC [email protected] >>>>> wrote: >>>>> >>>>>> Hi Julius, >>>>>> >>>>>> Please find my recording rules file attached. There isn't any rules >>>>>> using same names. >>>>>> Please also check it? Where i am doing wrong? >>>>>> >>>>>> Thanks >>>>>> >>>>>> 13 Aralık 2022 Salı tarihinde saat 12:07:31 UTC+3 itibarıyla >>>>>> [email protected] şunları yazdı: >>>>>> >>>>>>> Hi Umut, >>>>>>> >>>>>>> This error occurs when something in Prometheus tries to append a >>>>>>> sample to a time series with the same timestamp as the last already >>>>>>> stored >>>>>>> sample in the series, but with a different sample value. Usually this >>>>>>> is >>>>>>> because of some accidental collision between time series identities >>>>>>> (metric >>>>>>> name + label set). Is there any chance that anything else in your >>>>>>> Prometheus setup also produces metrics with the same names as the ones >>>>>>> in >>>>>>> your recording rules? Maybe the recording rules are duplicated, but >>>>>>> with >>>>>>> different values (probably in the same rule group, since it's the same >>>>>>> timestamp)? >>>>>>> >>>>>>> Regards, >>>>>>> Julius >>>>>>> >>>>>>> On Tue, Dec 13, 2022 at 6:54 AM Umut Cokbilir <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> I couldn't understand the root cause of the problem below. How to >>>>>>>> solve it? >>>>>>>> >>>>>>>> Prometheus version: prometheus-2.40.3.linux-amd64 >>>>>>>> snmp_exporter version: snmp_exporter-0.20.0 >>>>>>>> >>>>>>>> Rules: >>>>>>>> - record: fix:min_in >>>>>>>> expr: min_over_time(access:ifHCInOctets:FIX_rate[1d]) >>>>>>>> - record: fix:min_out >>>>>>>> expr: min_over_time(access:ifHCOutOctets:FIX_rate[1d]) >>>>>>>> - record: fix:max_in >>>>>>>> expr: max_over_time(access:ifHCInOctets:FIX_rate[1d]) >>>>>>>> - record: fix:max_out >>>>>>>> expr: max_over_time(access:ifHCOutOctets:FIX_rate[1d]) >>>>>>>> - record: fix:avg_in >>>>>>>> expr: avg_over_time(access:ifHCInOctets:FIX_rate[1d]) >>>>>>>> - record: fix:avg_out >>>>>>>> expr: avg_over_time(access:ifHCOutOctets:FIX_rate[1d]) >>>>>>>> - record: fix:avg_in:12h >>>>>>>> expr: avg_over_time(access:ifHCInOctets:FIX_rate[12h]) >>>>>>>> - record: fix:avg_out:12h >>>>>>>> expr: avg_over_time(access:ifHCOutOctets:FIX_rate[12h]) >>>>>>>> >>>>>>>> 04:41:23 --ts=2022-12-13T01:41:23.771Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:min_in >>>>>>>> index=16 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>> for >>>>>>>> timestamp" sample="{__name__=\"fix:min_in\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 22676.373333333333 @[1670895366553]" >>>>>>>> 04:41:25 --ts=2022-12-13T01:41:25.099Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in >>>>>>>> index=20 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>> for >>>>>>>> timestamp" sample="{__name__=\"fix:avg_in\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 45519.09881820317 @[1670895366553]" >>>>>>>> 04:41:25 --ts=2022-12-13T01:41:25.495Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out >>>>>>>> index=21 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>> for >>>>>>>> timestamp" sample="{__name__=\"fix:avg_out\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 571800.1416907905 @[1670895366553]" >>>>>>>> 04:41:26 --ts=2022-12-13T01:41:25.864Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>> name=fix:avg_in:12h >>>>>>>> index=22 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>> for >>>>>>>> timestamp" sample="{__name__=\"fix:avg_in:12h\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 37126.089073190036 @[1670895366553]" >>>>>>>> 04:41:26 --ts=2022-12-13T01:41:26.208Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>> name=fix:avg_out:12h >>>>>>>> index=23 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>> for >>>>>>>> timestamp" sample="{__name__=\"fix:avg_out:12h\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 568421.9923866978 @[1670895366553]" >>>>>>>> 05:21:23 --ts=2022-12-13T02:21:23.715Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in >>>>>>>> index=20 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>> for >>>>>>>> timestamp" sample="{__name__=\"fix:avg_in\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 45353.12701316737 @[1670897766553]" >>>>>>>> 05:21:24 --ts=2022-12-13T02:21:24.126Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out >>>>>>>> index=21 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>> for >>>>>>>> timestamp" sample="{__name__=\"fix:avg_out\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 571748.0096685543 @[1670897766553]" >>>>>>>> 05:21:24 --ts=2022-12-13T02:21:24.453Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>> name=fix:avg_in:12h >>>>>>>> index=22 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>> for >>>>>>>> timestamp" sample="{__name__=\"fix:avg_in:12h\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 31094.769935320466 @[1670897766553]" >>>>>>>> 05:21:25 --ts=2022-12-13T02:21:24.794Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>> name=fix:avg_out:12h >>>>>>>> index=23 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>> for >>>>>>>> timestamp" sample="{__name__=\"fix:avg_out:12h\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 567518.4603637857 @[1670897766553]" >>>>>>>> 06:06:26 --ts=2022-12-13T03:06:25.984Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in >>>>>>>> index=20 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>> for >>>>>>>> timestamp" sample="{__name__=\"fix:avg_in\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 45189.166572681956 @[1670900466553]" >>>>>>>> 06:06:26 --ts=2022-12-13T03:06:26.353Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out >>>>>>>> index=21 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>> for >>>>>>>> timestamp" sample="{__name__=\"fix:avg_out\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 571701.6636125795 @[1670900466553]" >>>>>>>> 06:06:26 --ts=2022-12-13T03:06:26.679Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>> name=fix:avg_in:12h >>>>>>>> index=22 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>> for >>>>>>>> timestamp" sample="{__name__=\"fix:avg_in:12h\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 29880.13207330078 @[1670900466553]" >>>>>>>> 06:06:27 --ts=2022-12-13T03:06:27.010Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>> name=fix:avg_out:12h >>>>>>>> index=23 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>> for >>>>>>>> timestamp" sample="{__name__=\"fix:avg_out:12h\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 567379.9435644317 @[1670900466553]" >>>>>>>> 06:21:27 --ts=2022-12-13T03:21:27.089Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in >>>>>>>> index=20 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>> for >>>>>>>> timestamp" sample="{__name__=\"fix:avg_in\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 45110.958791205536 @[1670901366553]" >>>>>>>> 06:21:27 --ts=2022-12-13T03:21:27.496Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out >>>>>>>> index=21 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>> for >>>>>>>> timestamp" sample="{__name__=\"fix:avg_out\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 571723.652811564 @[1670901366553]" >>>>>>>> 06:21:27 --ts=2022-12-13T03:21:27.832Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>> name=fix:avg_in:12h >>>>>>>> index=22 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>> for >>>>>>>> timestamp" sample="{__name__=\"fix:avg_in:12h\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 29745.59755724392 @[1670901366553]" >>>>>>>> 06:21:28 --ts=2022-12-13T03:21:28.173Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>> name=fix:avg_out:12h >>>>>>>> index=23 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>> for >>>>>>>> timestamp" sample="{__name__=\"fix:avg_out:12h\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 567638.6622438562 @[1670901366553]" >>>>>>>> 06:56:28 --ts=2022-12-13T03:56:28.058Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in >>>>>>>> index=20 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>> for >>>>>>>> timestamp" sample="{__name__=\"fix:avg_in\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 44958.979497887565 @[1670903466553]" >>>>>>>> 06:56:28 --ts=2022-12-13T03:56:28.650Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out >>>>>>>> index=21 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>> for >>>>>>>> timestamp" sample="{__name__=\"fix:avg_out\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 571641.2677371124 @[1670903466553]" >>>>>>>> 06:56:29 --ts=2022-12-13T03:56:29.089Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>> name=fix:avg_in:12h >>>>>>>> index=22 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>> for >>>>>>>> timestamp" sample="{__name__=\"fix:avg_in:12h\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 29316.18369902085 @[1670903466553]" >>>>>>>> 06:56:29 --ts=2022-12-13T03:56:29.713Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>> name=fix:avg_out:12h >>>>>>>> index=23 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>> for >>>>>>>> timestamp" sample="{__name__=\"fix:avg_out:12h\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 567614.2432218753 @[1670903466553]" >>>>>>>> 07:06:08 --ts=2022-12-13T04:06:07.854Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>> name=access:ifHCInOctets:rate index=0 msg="Rule evaluation result >>>>>>>> discarded" err="duplicate sample for timestamp" >>>>>>>> sample="{__name__=\"access:ifHCInOctets:rate\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 26498.213333333337 @[1670904066553]" >>>>>>>> 07:06:10 --ts=2022-12-13T04:06:09.708Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>> name=access:ifHCOutOctets:rate index=1 msg="Rule evaluation result >>>>>>>> discarded" err="duplicate sample for timestamp" >>>>>>>> sample="{__name__=\"access:ifHCOutOctets:rate\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 569222.2666666667 @[1670904066553]" >>>>>>>> 07:06:12 --ts=2022-12-13T04:06:11.181Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>> name=access:ifHCInOctets:FIX_rate index=4 msg="Rule evaluation result >>>>>>>> discarded" err="duplicate sample for timestamp" >>>>>>>> sample="{__name__=\"access:ifHCInOctets:FIX_rate\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 26498.213333333337 @[1670904066553]" >>>>>>>> 07:06:13 --ts=2022-12-13T04:06:11.676Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>> name=access:ifHCOutOctets:FIX_rate index=5 msg="Rule evaluation result >>>>>>>> discarded" err="duplicate sample for timestamp" >>>>>>>> sample="{__name__=\"access:ifHCOutOctets:FIX_rate\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 569222.2666666667 @[1670904066553]" >>>>>>>> 07:51:27 --ts=2022-12-13T04:51:27.065Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in >>>>>>>> index=20 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>> for >>>>>>>> timestamp" sample="{__name__=\"fix:avg_in\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 44893.41348050907 @[1670906766553]" >>>>>>>> 07:51:27 --ts=2022-12-13T04:51:27.564Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out >>>>>>>> index=21 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>> for >>>>>>>> timestamp" sample="{__name__=\"fix:avg_out\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 571623.9997716311 @[1670906766553]" >>>>>>>> 07:51:28 --ts=2022-12-13T04:51:27.907Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>> name=fix:avg_in:12h >>>>>>>> index=22 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>> for >>>>>>>> timestamp" sample="{__name__=\"fix:avg_in:12h\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 28273.031953867194 @[1670906766553]" >>>>>>>> 07:51:28 --ts=2022-12-13T04:51:28.245Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>> name=fix:avg_out:12h >>>>>>>> index=23 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>> for >>>>>>>> timestamp" sample="{__name__=\"fix:avg_out:12h\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 568057.166386401 @[1670906766553]" >>>>>>>> 08:01:08 --ts=2022-12-13T05:01:08.140Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>> name=access:ifHCInOctets:rate index=0 msg="Rule evaluation result >>>>>>>> discarded" err="duplicate sample for timestamp" >>>>>>>> sample="{__name__=\"access:ifHCInOctets:rate\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 26605.573333333334 @[1670907366553]" >>>>>>>> 08:01:10 --ts=2022-12-13T05:01:09.841Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>> name=access:ifHCOutOctets:rate index=1 msg="Rule evaluation result >>>>>>>> discarded" err="duplicate sample for timestamp" >>>>>>>> sample="{__name__=\"access:ifHCOutOctets:rate\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 566597.4400000001 @[1670907366553]" >>>>>>>> 08:01:12 --ts=2022-12-13T05:01:11.039Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>> name=access:ifHCInOctets:FIX_rate index=4 msg="Rule evaluation result >>>>>>>> discarded" err="duplicate sample for timestamp" >>>>>>>> sample="{__name__=\"access:ifHCInOctets:FIX_rate\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 26605.573333333334 @[1670907366553]" >>>>>>>> 08:01:13 --ts=2022-12-13T05:01:11.295Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>> name=access:ifHCOutOctets:FIX_rate index=5 msg="Rule evaluation result >>>>>>>> discarded" err="duplicate sample for timestamp" >>>>>>>> sample="{__name__=\"access:ifHCOutOctets:FIX_rate\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 566597.4400000001 @[1670907366553]" >>>>>>>> 08:16:26 --ts=2022-12-13T05:16:26.233Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in >>>>>>>> index=20 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>> for >>>>>>>> timestamp" sample="{__name__=\"fix:avg_in\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 44828.267673529954 @[1670908266553]" >>>>>>>> 08:16:26 --ts=2022-12-13T05:16:26.661Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out >>>>>>>> index=21 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>> for >>>>>>>> timestamp" sample="{__name__=\"fix:avg_out\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 571600.3475860697 @[1670908266553]" >>>>>>>> 08:16:27 --ts=2022-12-13T05:16:27.003Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>> name=fix:avg_in:12h >>>>>>>> index=22 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>> for >>>>>>>> timestamp" sample="{__name__=\"fix:avg_in:12h\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 27422.084942333982 @[1670908266553]" >>>>>>>> 08:16:27 --ts=2022-12-13T05:16:27.338Z caller=manager.go:684 >>>>>>>> level=warn component="rule manager" >>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>> name=fix:avg_out:12h >>>>>>>> index=23 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>> for >>>>>>>> timestamp" sample="{__name__=\"fix:avg_out:12h\", >>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>> instance=\"10.85.4.3\", >>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>> 568112.2042330011 @[1670908266553]" >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Prometheus Users" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/prometheus-users/0f1c9697-8c58-4895-8917-182a66ed217dn%40googlegroups.com >>>>>>>> >>>>>>>> <https://groups.google.com/d/msgid/prometheus-users/0f1c9697-8c58-4895-8917-182a66ed217dn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Julius Volz >>>>>>> PromLabs - promlabs.com >>>>>>> >>>>>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/05123d4b-11de-41b8-bbf0-d8332b6b76ebn%40googlegroups.com.
