How long do you usually wait before restarting the Prometheus process? https://www.robustperception.io/debugging-out-of-order-samples
There are a couple of different possible root cause so I'm 100% sure which one you are hitting. Prometheus itself might not pick up the time change to the correct one once the VM has resumed. It might take multiple scraping cycles for all targets. You can check the "prometheus_target_scrapes_sample_out_of_order_total" counter or "timestamp(up)" do see what issue you are hitting. Either way, a server shutdown + start is not too bad ;) On Sat, Nov 7, 2020 at 7:48 PM Harald Koch <[email protected]> wrote: > On Thu, Nov 5, 2020, at 03:54, Ben Kochie wrote: > > For working hypervisor combinations, running NTP on the guest VM is > unnecessary. For example, with KVM/QEMU, the guest can sync directly with > the hypervisor. > > > Thanks for your reply! > > I agree, but the the state of the art continues to be that this ... > doesn't work very well. Even RedHat's KVM+QEMU documentation recommends > that the guests run NTP. > > I'm still curious what the source of the issue is. Laurent Dumont > mentioned that the clocks on the scraped target and the prometheus host > will be different, but based on what I've seen, there are no timestamps in > the exporter outputs, so why would this matter? > > Even with NTPD running, the libvirtd resume operation resets the clock in > the guest (I'm running the qemu-agent in the guest, and have the > SYNC_TIME=1 option set on the host). The reboot takes about 2 minutes, so > even if some samples get scraped into the TSDB with strange future > timestamps, shouldn't this should correct after a few minutes, once the > prometheus host catches back up to the present? > > And the strange part for me - restarting prometheus immediately fixes the > issue. If there were future-dated samples collected, wouldn't the problem > persist after restarting the daemon? > > Still confused, > > -- > Harald > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAOAKi8x1QB36p%3DPFJQOfCkML5ze0Pe5p_ZWTXYwsrztSLK10gw%40mail.gmail.com.
