I'm not sure if I fully understand the flow of metrics, but you can use a PushGateway as a "central" scrape target. In your case, since your application pods seem to be short-lived, a per pod scrape architecture might not be great. You can use the prometheus_client library to push application metrics to the PushGateway pod which is then scraped by Prometheus.
The PushGateway will always be online and available as a target to Prometheus. On Thu, Nov 12, 2020 at 8:47 AM Rafael Paulovic <[email protected]> wrote: > Hi all, I have an architectural question. > > I am using Prometheus as follows: > > In a K8S cluster, running pods with multiple running processes running in > a single container. These processes send metrics via a websocket connection > to a central process which exposes a /metrics endpoint so that Prometheus > scrapes from there the metrics from all processes. > > When all process are finished they might send some special metrics just > before finishing. > > But all when all these process finish, this also makes the pod goes to > completed, closing the http server and therefore making Prometheus unable > to scrape. > > What is the best way to guarantee that the latest metrics sent are scraped? > > Would I need to wait that Prometheus scrapes the last metric before pod > goes to completed? > > I was planning to do that, but then I saw that the interval in the cluster > is too big (1 minute + potentially the time to discover the job). > > Does anyone has another, more feasible idea? Am I missing something? > > Thanks, > Best regards, > Rafael. > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/f0b0cc7a-377a-4c0d-9e38-3d36f1d9ae9dn%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/f0b0cc7a-377a-4c0d-9e38-3d36f1d9ae9dn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAOAKi8w%2Bn6ftV5WUeatcd5cWUQkSvDkBgOf3z8M3nRtzPp%2BxfQ%40mail.gmail.com.
