I have a server in a Kubernetes Pod that handles requests. It might happen that a request is issued before the Pod has started (for instance if the Pod fails and gets re-created). In these cases, the request is queued by an external system and delivered to the Pod as soon as it starts.
I want to track, for requests that were issued before the Pod started, the time from the instant the request is sent to the time the request is received by the Pod via a Prometheus Histogram. The gist of the problem is that all the observations in such Histogram happen right after the Pod has started, because the Pod receives all the requests that piled up while it was not running at start up. This means that the histogram has all of its observations by the time the first Prometheus scrape happens; the consequence is that PromQL `rate()` completely fails to see the change in the histogram, and this makes it impossible to make useful plots. Is there a known solution for this problem? I thought about making the server sleep at least as long as Prometheus' scrape interval right after starting to serve metrics to ensure that there is at least one scrape that sees the histogram with no observations, but that is super-hackish, I'd rather do something else. Thanks, Matteo. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/7cb031f5-6cc0-467b-8a7e-1df92adf94b4%40googlegroups.com.
