For this use case, it's likely what they want is Prometheus in agent mode, which uses remote write, which can buffer and catch up.
On Sat, Jun 18, 2022, 1:13 PM Stuart Clark <[email protected]> wrote: > On 14/06/2022 18:32, Jeremy Collette wrote: > > Hello, > > We have written a custom exporter that exposes metrics with explicit > timestamps, which Prometheus periodically scrapes. In the case where > Prometheus becomes temporarily unavailable, these metric samples will be > cached in the exporter until they are scraped, causing affected metrics to > age. > > I understand that if a metric is older than a certain threshold, it will > be rejected by Prometheus with the message: "Error on ingesting samples > that are too old or are too far into the future". > > I'm trying to understand if there are any guarantees surrounding the > ingestion of historical metrics. Is there some metric sample age that is > guaranteed to be recent enough to be ingested? For example, are samples > with timestamps within the last hour always going to be considered recent? > Within the last five minutes? > > According to this previous thread: Error on ingesting samples that are > too old > <https://groups.google.com/g/prometheus-users/c/rKJYm6naEow/m/zylud_J4AAAJ>, > MR seems to indicate that metrics as old as 1 second can be dropped due to > being too old. Is this interpretation correct? If so, is there any way to > ensure metrics with timestamps won't be dropped for being too old? > > The use of timestamps in metrics is not something that should be used > except in some very specific cases. The main use case for adding a > timestamp is when you are scraping metrics into Prometheus that have been > sourced from another existing metrics system (for example things like the > Cloudwatch Exporter). You also mention something about your exporter > caching things until they are scraped, which also sounds like something > that is not advisable. The action of the exporter shouldn't really be > changing depending on the requests being received (or not received). > > An exporter is expected to return the various metrics that reflect "now", > in the same way that a directly instrumented application would be expected > to return the current state of the metrics being maintained in memory. For > a simple exporter the normal mechanism is for a request to be received > which then triggers some mechanism to generate the metrics. For example > with something like the MySQL Exporter a request would trigger a query on > the connected database which then returns various information that is > converted into Prometheus metrics and returned. In some situations the > process to fetch information from the underlying system can be quite > resource intensive or slow. In that case a common design is to decouple the > information fetching process from the request handling process. One example > is to perform the information fetching process on a periodic timer, with > the information fetched then stored in memory. The request process then > reads and returns that information - returning the same values for every > request until the next cycle of the information fetching process. In none > of these standard scenarios would you expect timestamps to be attached to > the returned metrics. > > It would be good to hear a bit more about what you are trying to do, as it > is highly likely that the use of timestamps in your use case is probably > not the right option and they should just be dropped. > > -- > Stuart Clark > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/007cf462-d87e-d4c7-316e-4007567c74a1%40Jahingo.com > <https://groups.google.com/d/msgid/prometheus-users/007cf462-d87e-d4c7-316e-4007567c74a1%40Jahingo.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmoHG%3DeEJbQXWM4QaN%2BDuwcBfzBLPNktSzcMLk6Q7oQViw%40mail.gmail.com.
