It can't (simplistic system which probably needs replacing in the next few months). This will still cause them to have gaps. Basically these tasks don't run very often (part of them run once a week), and as we update the platform often some of these tasks run less often then a deploy. It basically sounds like I need to record these into my own database and export those to prometheus?
On Thursday, 27 February 2020 14:45:42 UTC+1, Stuart Clark wrote: > > Can the system which controls the tasks collect the information when a > task finishes and then add it to metrics it exposes? Total number of tasks > completed, total time spent, etc. > > On 27 February 2020 13:29:28 GMT, Paul van der Linden < > [email protected] <javascript:>> wrote: >> >> I'm trying to scrape metrics from tasks. The current task system spins up >> a process per task, which means I can't really get these scraped by >> prometheus (most of these metrics are produced at the end of the tasks >> (like running time of the task), so they would disappear immediately after >> they are available. Instead I have a sidecar(kubernetes) pushgateway >> running. This solves the issue partially. But when I have an update, the >> pod will be replaced, and the pushgateway would also disappear. This means >> it's difficult to see differences in these metrics as they are spread out >> and disconnected (gaps in the line). If I run a single pushgateway I would >> have metrics sticking around forever, and would have to clean these up if >> something changes. What would be the best solution for recording metrics on >> jobs/tasks (processes/pods which disappear quickly after (some of) the >> metrics became available), without running into these issues? >> >> > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ea3b4e52-028d-44da-bb1a-423704f82d52%40googlegroups.com.
