For working hypervisor combinations, running NTP on the guest VM is unnecessary. For example, with KVM/QEMU, the guest can sync directly with the hypervisor.
$ cat /sys/devices/system/clocksource/clocksource0/current_clocksource kvm-clock https://www.linux-kvm.org/page/KVMClock On Thu, Nov 5, 2020 at 3:27 AM Laurent Dumont <[email protected]> wrote: > When you resume the VM, does NTP resync? I guess that since you just pause > the VM, once it unpause, the time will be incorrect and that would explain > the out-of-order message. The scaped host will be in the future from the > perspective of the Prometheus server. > > On Wed, Nov 4, 2020 at 8:46 PM Harald Koch <[email protected]> wrote: > >> I'm running prometheus at home, to monitor a few VMs on my homelab, as >> well as my Linodes. My homelab hypervisor runs Arch Linux and libvirtd. >> >> Sometimes I need to patch the OS on the hypervisor and reboot it. The >> process up until now has been to suspend the guests, reboot the server, and >> restore the guests. (Yeah, I know, this is not recommended for prometheus - >> but it's a homelab, not a production environment . And I've seen weird >> clock-skew errors on production VMware before... :). >> >> This has been working for a couple of years at this point. But recently >> (like, the last month or so) I've been getting prometheus "Error on >> ingesting out-of-order samples" errors spewing in the logs after this >> operation, and no data stored in prometheus. Restarting the prometheus >> process fixes it immediately; the errors stop and data is stored in the >> database again. >> >> The problem has something to do with the combination of >> suspending/resuming the VM, and clock synchronization (hypervisor -> guest, >> or NTP is unknown). Looking for advice on how to debug the problem (or >> someone who has already encountered this...) >> >> Thanks! >> >> -- >> Harald >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/5a03f52b-c718-4b05-99e6-4330fb67608d%40www.fastmail.com >> <https://groups.google.com/d/msgid/prometheus-users/5a03f52b-c718-4b05-99e6-4330fb67608d%40www.fastmail.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/CAOAKi8yqTXQthunTsh5pGr2Yqu8XTtYFzjhWNk6xGtaCe19Ndw%40mail.gmail.com > <https://groups.google.com/d/msgid/prometheus-users/CAOAKi8yqTXQthunTsh5pGr2Yqu8XTtYFzjhWNk6xGtaCe19Ndw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmqo2X%2Bn2LsvommDgfhutvaCiUXQpmUhktdwSM2RisoD%3Dg%40mail.gmail.com.
