Thank you for the full files (although you probably should have scrubbed
your snmp community)
The only slightly suspicious thing I see is
honor_timestamps: true
which generally shouldn't be needed except in very special circumstances.
I don't *think* it will make any difference here, since snmp_exporter
doesn't give timestamps with its metrics anyway, but can you try commenting
it out?
Otherwise, follow Julien's advice of trying the same configuration with
v2.37.5 (which is an LTS release) and see if the problem recurs.
On Tuesday, 13 December 2022 at 15:28:37 UTC [email protected] wrote:
> Hi All,
>
> I added my all config files attached.
> Thank you so much for this deep diving.
>
> 13 Aralık 2022 Salı tarihinde saat 17:37:43 UTC+3 itibarıyla Brian Candler
> şunları yazdı:
>
>> I ran your curl output through "promtool check metrics", but it only
>> warns about label and metric naming conventions (which SNMP violates to be
>> consistent with standard MIB usage). There are no duplicate timeseries.
>>
>> # /opt/prometheus/promtool check metrics <ert3.txt 2>&1 | sort | uniq -c
>> 11247 ifAlias label names should be written in 'snake_case' not
>> 'camelCase'
>> 1 ifAlias metric names should be written in 'snake_case' not
>> 'camelCase'
>> 13767 ifDescr label names should be written in 'snake_case' not
>> 'camelCase'
>> 1 ifDescr metric names should be written in 'snake_case' not
>> 'camelCase'
>> 1 ifHCInOctets counter metrics should have "_total" suffix
>> 11247 ifHCInOctets label names should be written in 'snake_case' not
>> 'camelCase'
>> 1 ifHCInOctets metric names should be written in 'snake_case' not
>> 'camelCase'
>> 1 ifHCOutOctets counter metrics should have "_total" suffix
>> 11247 ifHCOutOctets label names should be written in 'snake_case' not
>> 'camelCase'
>> 1 ifHCOutOctets metric names should be written in 'snake_case' not
>> 'camelCase'
>> 1 sysName label names should be written in 'snake_case' not
>> 'camelCase'
>> 1 sysName metric names should be written in 'snake_case' not
>> 'camelCase'
>> 1 sysUpTime metric names should be written in 'snake_case' not
>> 'camelCase'
>>
>> Changing the target.yml file periodically won't make a difference;
>> Prometheus will just pick up the new one for the next scrape. So this is
>> odd.
>>
>> On Tuesday, 13 December 2022 at 12:21:15 UTC [email protected] wrote:
>>
>>> Thanks for your detail explanation. You can see my answers below.
>>>
>>> I am not doing federation or any sort of availability.
>>> I've checked [1h] then [1d] and haven't seen any multiple data points :(
>>> I am not scraping the same targets twice but i am updating target.yml
>>> every hour. Does it affect?
>>> i am not sure there is any bug in my exporter config but it scrapes data
>>> from mydevices every 5 minutes.
>>> Please find curl output attached.
>>>
>>>
>>> 13 Aralık 2022 Salı tarihinde saat 14:44:31 UTC+3 itibarıyla Brian
>>> Candler şunları yazdı:
>>>
>>>> The first error explicitly mentions a particular metric "fix:min_in",
>>>> and the input to the recording rule which generates that is the metric
>>>> "access:ifHCInOctets:FIX_rate" (rule 16, counting from zero)
>>>> But it also mentions an error for the metric
>>>> "access:ifHCInOctets:FIX_rate", which takes its input from "ifHCInOctets"
>>>> (rule 4)
>>>>
>>>> So my guess is that you have duplicate data somewhere, posisbly in
>>>> ifHCInOctets. Try these queries:
>>>>
>>>>
>>>> ifHCInOctets{instance="10.85.4.3",ifAlias="FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||"}[1h]
>>>>
>>>> access:ifHCInOctets:FIX_rate{instance="10.85.4.3",ifAlias="FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||"}[1h]
>>>>
>>>> Do you see multiple data points with the same timestamp? In which
>>>> timeseries? if not, try extending the time range to [1d]
>>>>
>>>> This would however require you to have multiple values of ifHCInOctets
>>>> with the same value of all labels (same ifAlias, same ifDescr, same
>>>> ifIndex).
>>>>
>>>> If that's what you're seeing then you have to work backwards: are you
>>>> scraping the same target twice? Or is there a bug in your snmp_exporter
>>>> config? Try scraping the device manually using curl:
>>>>
>>>> curl -g 'localhost:9115/snmp?module=XXXX&target=10.85.4.3'
>>>>
>>>> (Having said that, if there were duplicates at this point, I'd be
>>>> expecting them to be rejected at scrape time).
>>>>
>>>> Are you doing any sort of high-availability or federation? If so, you
>>>> need different external labels for each data source.
>>>>
>>>> On Tuesday, 13 December 2022 at 10:55:19 UTC [email protected] wrote:
>>>>
>>>>> Hi Julius,
>>>>>
>>>>> Please find my recording rules file attached. There isn't any rules
>>>>> using same names.
>>>>> Please also check it? Where i am doing wrong?
>>>>>
>>>>> Thanks
>>>>>
>>>>> 13 Aralık 2022 Salı tarihinde saat 12:07:31 UTC+3 itibarıyla
>>>>> [email protected] şunları yazdı:
>>>>>
>>>>>> Hi Umut,
>>>>>>
>>>>>> This error occurs when something in Prometheus tries to append a
>>>>>> sample to a time series with the same timestamp as the last already
>>>>>> stored
>>>>>> sample in the series, but with a different sample value. Usually this is
>>>>>> because of some accidental collision between time series identities
>>>>>> (metric
>>>>>> name + label set). Is there any chance that anything else in your
>>>>>> Prometheus setup also produces metrics with the same names as the ones
>>>>>> in
>>>>>> your recording rules? Maybe the recording rules are duplicated, but with
>>>>>> different values (probably in the same rule group, since it's the same
>>>>>> timestamp)?
>>>>>>
>>>>>> Regards,
>>>>>> Julius
>>>>>>
>>>>>> On Tue, Dec 13, 2022 at 6:54 AM Umut Cokbilir <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> I couldn't understand the root cause of the problem below. How to
>>>>>>> solve it?
>>>>>>>
>>>>>>> Prometheus version: prometheus-2.40.3.linux-amd64
>>>>>>> snmp_exporter version: snmp_exporter-0.20.0
>>>>>>>
>>>>>>> Rules:
>>>>>>> - record: fix:min_in
>>>>>>> expr: min_over_time(access:ifHCInOctets:FIX_rate[1d])
>>>>>>> - record: fix:min_out
>>>>>>> expr: min_over_time(access:ifHCOutOctets:FIX_rate[1d])
>>>>>>> - record: fix:max_in
>>>>>>> expr: max_over_time(access:ifHCInOctets:FIX_rate[1d])
>>>>>>> - record: fix:max_out
>>>>>>> expr: max_over_time(access:ifHCOutOctets:FIX_rate[1d])
>>>>>>> - record: fix:avg_in
>>>>>>> expr: avg_over_time(access:ifHCInOctets:FIX_rate[1d])
>>>>>>> - record: fix:avg_out
>>>>>>> expr: avg_over_time(access:ifHCOutOctets:FIX_rate[1d])
>>>>>>> - record: fix:avg_in:12h
>>>>>>> expr: avg_over_time(access:ifHCInOctets:FIX_rate[12h])
>>>>>>> - record: fix:avg_out:12h
>>>>>>> expr: avg_over_time(access:ifHCOutOctets:FIX_rate[12h])
>>>>>>>
>>>>>>> 04:41:23 --ts=2022-12-13T01:41:23.771Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:min_in
>>>>>>> index=16 msg="Rule evaluation result discarded" err="duplicate sample
>>>>>>> for
>>>>>>> timestamp" sample="{__name__=\"fix:min_in\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 22676.373333333333 @[1670895366553]"
>>>>>>> 04:41:25 --ts=2022-12-13T01:41:25.099Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in
>>>>>>> index=20 msg="Rule evaluation result discarded" err="duplicate sample
>>>>>>> for
>>>>>>> timestamp" sample="{__name__=\"fix:avg_in\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 45519.09881820317 @[1670895366553]"
>>>>>>> 04:41:25 --ts=2022-12-13T01:41:25.495Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out
>>>>>>> index=21 msg="Rule evaluation result discarded" err="duplicate sample
>>>>>>> for
>>>>>>> timestamp" sample="{__name__=\"fix:avg_out\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 571800.1416907905 @[1670895366553]"
>>>>>>> 04:41:26 --ts=2022-12-13T01:41:25.864Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw
>>>>>>> name=fix:avg_in:12h
>>>>>>> index=22 msg="Rule evaluation result discarded" err="duplicate sample
>>>>>>> for
>>>>>>> timestamp" sample="{__name__=\"fix:avg_in:12h\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 37126.089073190036 @[1670895366553]"
>>>>>>> 04:41:26 --ts=2022-12-13T01:41:26.208Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw
>>>>>>> name=fix:avg_out:12h
>>>>>>> index=23 msg="Rule evaluation result discarded" err="duplicate sample
>>>>>>> for
>>>>>>> timestamp" sample="{__name__=\"fix:avg_out:12h\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 568421.9923866978 @[1670895366553]"
>>>>>>> 05:21:23 --ts=2022-12-13T02:21:23.715Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in
>>>>>>> index=20 msg="Rule evaluation result discarded" err="duplicate sample
>>>>>>> for
>>>>>>> timestamp" sample="{__name__=\"fix:avg_in\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 45353.12701316737 @[1670897766553]"
>>>>>>> 05:21:24 --ts=2022-12-13T02:21:24.126Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out
>>>>>>> index=21 msg="Rule evaluation result discarded" err="duplicate sample
>>>>>>> for
>>>>>>> timestamp" sample="{__name__=\"fix:avg_out\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 571748.0096685543 @[1670897766553]"
>>>>>>> 05:21:24 --ts=2022-12-13T02:21:24.453Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw
>>>>>>> name=fix:avg_in:12h
>>>>>>> index=22 msg="Rule evaluation result discarded" err="duplicate sample
>>>>>>> for
>>>>>>> timestamp" sample="{__name__=\"fix:avg_in:12h\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 31094.769935320466 @[1670897766553]"
>>>>>>> 05:21:25 --ts=2022-12-13T02:21:24.794Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw
>>>>>>> name=fix:avg_out:12h
>>>>>>> index=23 msg="Rule evaluation result discarded" err="duplicate sample
>>>>>>> for
>>>>>>> timestamp" sample="{__name__=\"fix:avg_out:12h\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 567518.4603637857 @[1670897766553]"
>>>>>>> 06:06:26 --ts=2022-12-13T03:06:25.984Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in
>>>>>>> index=20 msg="Rule evaluation result discarded" err="duplicate sample
>>>>>>> for
>>>>>>> timestamp" sample="{__name__=\"fix:avg_in\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 45189.166572681956 @[1670900466553]"
>>>>>>> 06:06:26 --ts=2022-12-13T03:06:26.353Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out
>>>>>>> index=21 msg="Rule evaluation result discarded" err="duplicate sample
>>>>>>> for
>>>>>>> timestamp" sample="{__name__=\"fix:avg_out\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 571701.6636125795 @[1670900466553]"
>>>>>>> 06:06:26 --ts=2022-12-13T03:06:26.679Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw
>>>>>>> name=fix:avg_in:12h
>>>>>>> index=22 msg="Rule evaluation result discarded" err="duplicate sample
>>>>>>> for
>>>>>>> timestamp" sample="{__name__=\"fix:avg_in:12h\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 29880.13207330078 @[1670900466553]"
>>>>>>> 06:06:27 --ts=2022-12-13T03:06:27.010Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw
>>>>>>> name=fix:avg_out:12h
>>>>>>> index=23 msg="Rule evaluation result discarded" err="duplicate sample
>>>>>>> for
>>>>>>> timestamp" sample="{__name__=\"fix:avg_out:12h\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 567379.9435644317 @[1670900466553]"
>>>>>>> 06:21:27 --ts=2022-12-13T03:21:27.089Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in
>>>>>>> index=20 msg="Rule evaluation result discarded" err="duplicate sample
>>>>>>> for
>>>>>>> timestamp" sample="{__name__=\"fix:avg_in\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 45110.958791205536 @[1670901366553]"
>>>>>>> 06:21:27 --ts=2022-12-13T03:21:27.496Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out
>>>>>>> index=21 msg="Rule evaluation result discarded" err="duplicate sample
>>>>>>> for
>>>>>>> timestamp" sample="{__name__=\"fix:avg_out\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 571723.652811564 @[1670901366553]"
>>>>>>> 06:21:27 --ts=2022-12-13T03:21:27.832Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw
>>>>>>> name=fix:avg_in:12h
>>>>>>> index=22 msg="Rule evaluation result discarded" err="duplicate sample
>>>>>>> for
>>>>>>> timestamp" sample="{__name__=\"fix:avg_in:12h\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 29745.59755724392 @[1670901366553]"
>>>>>>> 06:21:28 --ts=2022-12-13T03:21:28.173Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw
>>>>>>> name=fix:avg_out:12h
>>>>>>> index=23 msg="Rule evaluation result discarded" err="duplicate sample
>>>>>>> for
>>>>>>> timestamp" sample="{__name__=\"fix:avg_out:12h\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 567638.6622438562 @[1670901366553]"
>>>>>>> 06:56:28 --ts=2022-12-13T03:56:28.058Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in
>>>>>>> index=20 msg="Rule evaluation result discarded" err="duplicate sample
>>>>>>> for
>>>>>>> timestamp" sample="{__name__=\"fix:avg_in\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 44958.979497887565 @[1670903466553]"
>>>>>>> 06:56:28 --ts=2022-12-13T03:56:28.650Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out
>>>>>>> index=21 msg="Rule evaluation result discarded" err="duplicate sample
>>>>>>> for
>>>>>>> timestamp" sample="{__name__=\"fix:avg_out\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 571641.2677371124 @[1670903466553]"
>>>>>>> 06:56:29 --ts=2022-12-13T03:56:29.089Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw
>>>>>>> name=fix:avg_in:12h
>>>>>>> index=22 msg="Rule evaluation result discarded" err="duplicate sample
>>>>>>> for
>>>>>>> timestamp" sample="{__name__=\"fix:avg_in:12h\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 29316.18369902085 @[1670903466553]"
>>>>>>> 06:56:29 --ts=2022-12-13T03:56:29.713Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw
>>>>>>> name=fix:avg_out:12h
>>>>>>> index=23 msg="Rule evaluation result discarded" err="duplicate sample
>>>>>>> for
>>>>>>> timestamp" sample="{__name__=\"fix:avg_out:12h\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 567614.2432218753 @[1670903466553]"
>>>>>>> 07:06:08 --ts=2022-12-13T04:06:07.854Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw
>>>>>>> name=access:ifHCInOctets:rate index=0 msg="Rule evaluation result
>>>>>>> discarded" err="duplicate sample for timestamp"
>>>>>>> sample="{__name__=\"access:ifHCInOctets:rate\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 26498.213333333337 @[1670904066553]"
>>>>>>> 07:06:10 --ts=2022-12-13T04:06:09.708Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw
>>>>>>> name=access:ifHCOutOctets:rate index=1 msg="Rule evaluation result
>>>>>>> discarded" err="duplicate sample for timestamp"
>>>>>>> sample="{__name__=\"access:ifHCOutOctets:rate\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 569222.2666666667 @[1670904066553]"
>>>>>>> 07:06:12 --ts=2022-12-13T04:06:11.181Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw
>>>>>>> name=access:ifHCInOctets:FIX_rate index=4 msg="Rule evaluation result
>>>>>>> discarded" err="duplicate sample for timestamp"
>>>>>>> sample="{__name__=\"access:ifHCInOctets:FIX_rate\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 26498.213333333337 @[1670904066553]"
>>>>>>> 07:06:13 --ts=2022-12-13T04:06:11.676Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw
>>>>>>> name=access:ifHCOutOctets:FIX_rate index=5 msg="Rule evaluation result
>>>>>>> discarded" err="duplicate sample for timestamp"
>>>>>>> sample="{__name__=\"access:ifHCOutOctets:FIX_rate\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 569222.2666666667 @[1670904066553]"
>>>>>>> 07:51:27 --ts=2022-12-13T04:51:27.065Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in
>>>>>>> index=20 msg="Rule evaluation result discarded" err="duplicate sample
>>>>>>> for
>>>>>>> timestamp" sample="{__name__=\"fix:avg_in\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 44893.41348050907 @[1670906766553]"
>>>>>>> 07:51:27 --ts=2022-12-13T04:51:27.564Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out
>>>>>>> index=21 msg="Rule evaluation result discarded" err="duplicate sample
>>>>>>> for
>>>>>>> timestamp" sample="{__name__=\"fix:avg_out\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 571623.9997716311 @[1670906766553]"
>>>>>>> 07:51:28 --ts=2022-12-13T04:51:27.907Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw
>>>>>>> name=fix:avg_in:12h
>>>>>>> index=22 msg="Rule evaluation result discarded" err="duplicate sample
>>>>>>> for
>>>>>>> timestamp" sample="{__name__=\"fix:avg_in:12h\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 28273.031953867194 @[1670906766553]"
>>>>>>> 07:51:28 --ts=2022-12-13T04:51:28.245Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw
>>>>>>> name=fix:avg_out:12h
>>>>>>> index=23 msg="Rule evaluation result discarded" err="duplicate sample
>>>>>>> for
>>>>>>> timestamp" sample="{__name__=\"fix:avg_out:12h\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 568057.166386401 @[1670906766553]"
>>>>>>> 08:01:08 --ts=2022-12-13T05:01:08.140Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw
>>>>>>> name=access:ifHCInOctets:rate index=0 msg="Rule evaluation result
>>>>>>> discarded" err="duplicate sample for timestamp"
>>>>>>> sample="{__name__=\"access:ifHCInOctets:rate\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 26605.573333333334 @[1670907366553]"
>>>>>>> 08:01:10 --ts=2022-12-13T05:01:09.841Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw
>>>>>>> name=access:ifHCOutOctets:rate index=1 msg="Rule evaluation result
>>>>>>> discarded" err="duplicate sample for timestamp"
>>>>>>> sample="{__name__=\"access:ifHCOutOctets:rate\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 566597.4400000001 @[1670907366553]"
>>>>>>> 08:01:12 --ts=2022-12-13T05:01:11.039Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw
>>>>>>> name=access:ifHCInOctets:FIX_rate index=4 msg="Rule evaluation result
>>>>>>> discarded" err="duplicate sample for timestamp"
>>>>>>> sample="{__name__=\"access:ifHCInOctets:FIX_rate\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 26605.573333333334 @[1670907366553]"
>>>>>>> 08:01:13 --ts=2022-12-13T05:01:11.295Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw
>>>>>>> name=access:ifHCOutOctets:FIX_rate index=5 msg="Rule evaluation result
>>>>>>> discarded" err="duplicate sample for timestamp"
>>>>>>> sample="{__name__=\"access:ifHCOutOctets:FIX_rate\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 566597.4400000001 @[1670907366553]"
>>>>>>> 08:16:26 --ts=2022-12-13T05:16:26.233Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in
>>>>>>> index=20 msg="Rule evaluation result discarded" err="duplicate sample
>>>>>>> for
>>>>>>> timestamp" sample="{__name__=\"fix:avg_in\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 44828.267673529954 @[1670908266553]"
>>>>>>> 08:16:26 --ts=2022-12-13T05:16:26.661Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out
>>>>>>> index=21 msg="Rule evaluation result discarded" err="duplicate sample
>>>>>>> for
>>>>>>> timestamp" sample="{__name__=\"fix:avg_out\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 571600.3475860697 @[1670908266553]"
>>>>>>> 08:16:27 --ts=2022-12-13T05:16:27.003Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw
>>>>>>> name=fix:avg_in:12h
>>>>>>> index=22 msg="Rule evaluation result discarded" err="duplicate sample
>>>>>>> for
>>>>>>> timestamp" sample="{__name__=\"fix:avg_in:12h\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 27422.084942333982 @[1670908266553]"
>>>>>>> 08:16:27 --ts=2022-12-13T05:16:27.338Z caller=manager.go:684
>>>>>>> level=warn component="rule manager"
>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw
>>>>>>> name=fix:avg_out:12h
>>>>>>> index=23 msg="Rule evaluation result discarded" err="duplicate sample
>>>>>>> for
>>>>>>> timestamp" sample="{__name__=\"fix:avg_out:12h\",
>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\",
>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\",
>>>>>>> instance=\"10.85.4.3\",
>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} =>
>>>>>>> 568112.2042330011 @[1670908266553]"
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "Prometheus Users" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to [email protected].
>>>>>>> To view this discussion on the web visit
>>>>>>> https://groups.google.com/d/msgid/prometheus-users/0f1c9697-8c58-4895-8917-182a66ed217dn%40googlegroups.com
>>>>>>>
>>>>>>> <https://groups.google.com/d/msgid/prometheus-users/0f1c9697-8c58-4895-8917-182a66ed217dn%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>> .
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Julius Volz
>>>>>> PromLabs - promlabs.com
>>>>>>
>>>>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/0cc5125b-e8ae-4f58-b57f-ab8f0517a663n%40googlegroups.com.
