I believe this is a fundamental characteristic of the TSDB. It appends data to a head chunk containing (I think) the last 2 hours of data. If you try to write data for "the future", you're trying to write into a chunk that doesn't exist yet.
I wouldn't expect the Prometheus project to expend effort and resources to support a use case which only applies where the user's own system is misconfigured. On Tuesday 19 December 2023 at 11:06:53 UTC Kesavanand Chavali wrote: > Yes, between GoLang exporter and Prometheus, we have set the > honorTimeStamp to true. > Thanos supports --tsdb.too-far-in-future.time-window to ingest metrics in > future. Is there a way Prometheus also supports that? Is there a plan for > that? > In our case, as the node is 1hr into past, if we correct the timestamp, > the metric will be 1hr into future for Prometheus... > > On Tue, Dec 19, 2023 at 3:14 PM 'Brian Candler' via Prometheus Users < > promethe...@googlegroups.com> wrote: > >> Hmm: are you saying that the problem is that Prometheus Agent is running >> on a system with the wrong clock, and Prometheus Agent is adding the wrong >> UTC timestamps to the scrapes, and then remote_write is carrying these >> wrong timestamps, and then the remote_write receiver is rejecting them for >> being in the future? >> >> Ergh. Trying to apply compensating timestamps by adding OpenMetrics >> timestamps at scrape time sounds like it's doomed to failure. I'm not sure >> how you would get Prometheus Agent to run in an environment where the clock >> is wrong. >> >> If they could arrange that the central Prometheus (with the correct >> clock) scrapes the exporter directly, via some sort of reverse tunnel if >> necessary, and get rid of Prometheus Agent entirely, that would work. >> >> But really, the solution is to fix the underlying clock or timezone issue. >> >> On Tuesday 19 December 2023 at 09:21:08 UTC Ben Kochie wrote: >> >>> I think the problem here is that they have the system clock set >>> incorrectly, intentionally. It's not tracking UTC with a local timezone >>> set. But it's tracking a local timezone and the system thinks that is UTC. >>> >>> So Prometheus thinks UTC is some random local timezone, not real UTC. >>> >>> For the record, Prometheus always uses UTC for timestamps. On Linux >>> systems Prometheus can figure out local time from UTC. Not sure about the >>> Windows behavior here. This might be a windows-specific UTC / timezone >>> handling issue. >>> >>> On Tue, Dec 19, 2023 at 8:59 AM 'Brian Candler' via Prometheus Users < >>> promethe...@googlegroups.com> wrote: >>> >>>> On Tuesday 19 December 2023 at 06:03:25 UTC Kesavanand Chavali wrote: >>>> >>>> We have a customer exporter written in GoLang that scrapes metrics from >>>> windows exporter and corrects the time to current UTC Time. This custom >>>> exporter is scrapped by Prometheus for every 4 minutes. >>>> >>>> >>>> Still makes no sense to me. Can you show some examples of the actual >>>> scrapes, e.g. tested using curl? >>>> >>>> - windows_exporter should not be adding timestamps to metrics (although >>>> I don't have a running instance to test with) so there should be nothing >>>> to >>>> change >>>> - your custom exporter should not be adding timestamps to metrics >>>> - prometheus by default records the scrape time, not the metric >>>> timestamp >>>> >>>> (Aside: there may be some metrics whose value *is* a timestamp, >>>> like node_boot_time_seconds in node_exporter. But a metric is just a >>>> number, so whether it's "in the future" or "in the past" makes no >>>> difference for that kind of metric) >>>> >>>> >>>> The custom exporter is written in GoLang and uses >>>> NewMetricWithTimestamp to add the correct timestamp >>>> >>>> >>>> Why are you doing this? Why not just use [Must]NewConstMetric? And why >>>> are you proxying through a custom exporter, rather than just having the >>>> agent scrape windows_exporter directly? >>>> >>>> >>>> If we scrape windows exporter directly from Prometheus and remote write >>>> to Thanos, Thanos accepts the metrics as Out of order ingestion is allowed. >>>> If we scrape our custom exporter then we see metrics too old or too >>>> future error in Prometheus logs >>>> >>>> >>>> Do you have "honor_timestamps: true" in Prometheus agent? If so, why? >>>> >>>> To me, it seems like you're swimming against the current here. Just do >>>> what Prometheus does by default, which is to set the timestamp of every >>>> scrape as the time when it was scraped. The state of the clock on the >>>> scrape target is irrelevant. >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Prometheus Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to prometheus-use...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/prometheus-users/89d0fb28-bc8d-43c0-9166-99e660cd9b7fn%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/prometheus-users/89d0fb28-bc8d-43c0-9166-99e660cd9b7fn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> > You received this message because you are subscribed to a topic in the >> Google Groups "Prometheus Users" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/prometheus-users/vtmeo06pxiE/unsubscribe >> . >> To unsubscribe from this group and all its topics, send an email to >> prometheus-use...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/0c96dce9-19f8-424e-b93e-50f49e8fd39bn%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/0c96dce9-19f8-424e-b93e-50f49e8fd39bn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Thanks and Regards, > Kesav > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a56ba96f-5778-40b9-8fe1-8cd1bd219b0an%40googlegroups.com.
