We've recently upgraded a fleet of machines to both Ubuntu 18.04 (Linux
5.3) and node-exporter to 1.0.1 (run under Docker). We're now seeing a lot
of messages like these:
CPU Idle counter jumped backwards, possible hotplug event, resetting CPU
stats
CPU User counter jumped backwards
CPU System counter jumped backwards
CPU Iowait counter jumped backwards
CPU SoftIRQ counter jumped backwards
I examined a handful of messages and the counter seems to go backwards by a
very small amount (0.01). Over the last 3 days, 74% of the messages are
from one machine and 24% from another, with multiple machines making up the
rest.
I assume that this something odd in the system itself and that
node-exporter is just reporting what it sees. But any tips on what might
cause it / how to fix it, and whether it is worth worrying about? Is it
likely to be caused by running under Docker?
Here's the uname and docker-compose.yml for completeness:
Linux imgr1 5.3.0-62-generic #56~18.04.1-Ubuntu SMP Wed Jun 24 16:17:03 UTC
2020 x86_64 x86_64 x86_64 GNU/Linux
version: '2'
services:
node-exporter:
image: prom/node-exporter:v1.0.1
command:
- '--collector.textfile.directory=/textfile'
- '--no-collector.zfs'
- '--no-collector.wifi'
- '--no-collector.bcache'
- '--path.rootfs=/host'
- '--collector.vmstat.fields=^(oom_kill|pgpg|pswp|nr|pg.*fault).*'
-
'--collector.filesystem.ignored-fs-types=^(tmpfs|autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|fuse.lxcfs|nfs|nsfs|ceph|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$$'
network_mode: host
pid: host
volumes:
- /var/spool/node-exporter:/textfile:ro
- /:/host:ro,rslave
Thanks
Bruce
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/2aac7b8b-8780-4a2f-973d-b0344720b876n%40googlegroups.com.
