Wouldn't that have the same issues? But thinking through it, it might just have 3 choices: keep it like it is and just accept the gaps in the lines, have a global one which then only forgets old metrics when restarted or maintained, or keep track of the metrics in my own database, and export them like normal.
Maybe it's useful if I describe the full use case: these are basically ETL like tasks on large datasets, we get updates on these datasets every week, and we rerun these jobs after configuration changes. Some of these jobs don't handle that much data and finish within a few seconds, but the larger jobs take a long time (up to several hours). Basically all these metrics are slow moving (a lot of them change once a week), in general it feels a bit wrong to use prometheus for this as the other metrics are actually scraped often and changed often. We only record basic metrics currently like status and duration, but it might be useful to keep track of more detailed metrics within the jobs. The main thing is: I currently use them to compare runs over time: did software changes actually improve the processing speed or not? Which might not actually fit prometheus. On Thursday, 27 February 2020 15:53:19 UTC+1, Brian Candler wrote: > > Would statsd_exporter be sufficient? (e.g. at the end of each job add 1 to > counters for jobs run/success/failed as appropriate; add the time spent to > a cumulative total time etc) > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/eaca4bba-2ace-465d-950f-aabb7c93e5c1%40googlegroups.com.
