I'm trying to scrape metrics from tasks. The current task system spins up a process per task, which means I can't really get these scraped by prometheus (most of these metrics are produced at the end of the tasks (like running time of the task), so they would disappear immediately after they are available. Instead I have a sidecar(kubernetes) pushgateway running. This solves the issue partially. But when I have an update, the pod will be replaced, and the pushgateway would also disappear. This means it's difficult to see differences in these metrics as they are spread out and disconnected (gaps in the line). If I run a single pushgateway I would have metrics sticking around forever, and would have to clean these up if something changes. What would be the best solution for recording metrics on jobs/tasks (processes/pods which disappear quickly after (some of) the metrics became available), without running into these issues?
-- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/001e5d66-8ea5-4138-840b-b6cbce20ef7d%40googlegroups.com.
