Interesting, thanks for the data. It does seem like the process is spending
a lot of time doing GC like I thought.
One trick you could try is to increase the memory allocated to the prober,
which would reduce the time spent on GC.
The default setting is is GOGC=100.
You could try increasing this by setting the environment variable, GOGC.
Try something like GOGC=200 or GOGC=300.
This will make the process use more memory, but it should reduce the CPU
time spent.
On Sun, Feb 25, 2024 at 11:18 PM Alexander Wilke <nachtfalk...@gmail.com>
wrote:
> Hello,
>
> I attached a few screenshots showing the results and graphs for 1h and 6h.
> In addition I added a screenshot from node_exporter metrics to give you an
> overview of the system itself.
> On the same system there is prometheus, grafana, snmp_exporter (200-800%
> CPU), smokeping prober, node_exporter, blackbox_exporter.
> The main CPU consumers are snmp_exporter and smokeping.
>
> Ben Kochie schrieb am Sonntag, 25. Februar 2024 um 19:22:35 UTC+1:
>
>> Looking at the CPU profile, I'm seeing almost all the time spent in the
>> Go runtime. Mostly the ICMP packet receiving code and garbage collection.
>> I'm not sure there's a lot we can optimize here as it's core Go code for
>> ICMP packet handling.
>>
>> Can you also post me a graph of a few metrics queries?
>>
>> rate(process_cpu_seconds_total{job="smokeping_prober"}[30s])
>> rate(go_gc_duration_seconds_count{job="smokeping_prober"}[5m])
>> rate(go_gc_duration_seconds_sum{job="smokeping_prober"}[5m])
>>
>>
>> On Sun, Feb 25, 2024 at 7:08 PM Alexander Wilke <nachtf...@gmail.com>
>> wrote:
>>
>>> Hello,
>>> any Chance to investigate the Reports and any suggestions?
>>>
>>> Alexander Wilke schrieb am Donnerstag, 22. Februar 2024 um 12:40:09
>>> UTC+1:
>>>
>>>> Hello,
>>>>
>>>> sorry for the delay. here are the results. to be honest - I do not
>>>> understand anything of it.
>>>>
>>>> Smokeping_Prober Heap:
>>>>
>>>>
>>>> https://pprof.me/a1e7400d32859dbc217e2182398485df/?profileType=profile%3Aalloc_objects%3Acount%3Aspace%3Abytes&dashboard_items=icicle
>>>>
>>>>
>>>>
>>>> smokeping_prober profile30s
>>>>
>>>>
>>>> https://pprof.me/340674b335e114e4b0df6b4582f0644e/?profileType=profile%3Asamples%3Acount%3Acpu%3Ananoseconds%3Adelta
>>>>
>>>> Ben Kochie schrieb am Dienstag, 20. Februar 2024 um 10:27:10 UTC+1:
>>>>
>>>>> Best thing you can do is capture some pprof data. That will show you
>>>>> what it's spending the time on.
>>>>>
>>>>> :9374/debug/pprof/heap
>>>>> :9374/debug/pprof/profile?seconds=30
>>>>>
>>>>> You can post the results to https://pprof.me/ for sharing.
>>>>>
>>>>> On Tue, Feb 20, 2024 at 6:22 AM Alexander Wilke <nachtf...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hello,
>>>>>> I am running smokeping_prober from one VM to Monitor around 500
>>>>>> destinations.
>>>>>> Around 30 devices are monitored with 0.2s Intervall and Others with
>>>>>> 1.65s Intervall.
>>>>>>
>>>>>> Prometheus scrapes every 5s.
>>>>>>
>>>>>> So there are roughly 600 icmp ipv4 24byte pings per Seconds.
>>>>>> CPU usage jumps between 700-1200% using "top"
>>>>>>
>>>>>> What Else except reducing Interval or Host Count could Help to reduce
>>>>>> CPU usage?
>>>>>> Is the UDP Socket "better" or any other optimization which could be
>>>>>> relevant for that Type of Traffic? Running on RHEL8
>>>>>>
>>>>>> Someone with similar CPU usage and this amount of pings per Seconds?
>>>>>> Maybe Others Ping 6.000 Destination every 10s?
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Prometheus Users" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to prometheus-use...@googlegroups.com.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/prometheus-users/d803c1a2-64ee-48d1-8513-b864856f53c8n%40googlegroups.com
>>>>>> <https://groups.google.com/d/msgid/prometheus-users/d803c1a2-64ee-48d1-8513-b864856f53c8n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>>
>>>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to prometheus-use...@googlegroups.com.
>>>
>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-users/62aeaacb-9fd6-4d64-8bec-d7171f592766n%40googlegroups.com
>>> <https://groups.google.com/d/msgid/prometheus-users/62aeaacb-9fd6-4d64-8bec-d7171f592766n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/045a6fb0-b82c-4425-8bae-e2c99403b2adn%40googlegroups.com
> <https://groups.google.com/d/msgid/prometheus-users/045a6fb0-b82c-4425-8bae-e2c99403b2adn%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/CABbyFmrDGh4cJWHCtmD_-SvJDEiTtA-kOnafmu0oe57F4rWz_w%40mail.gmail.com.
