You can also use a Prometheus long-term store that supports direct push. Promscale is one such LTS (https://github.com/timescale/promscale/blob/master/docs/writing_to_promscale.md), and the one I work at, but I believe there are others. But as MR points out the analysis/query side gets complicated in PromQL because there is an implicit assumption of regular intervals. Promscale's answer is to use SQL but this does complicate the analysis somewhat.
-- Mat Arye, Promscale <https://github.com/timescale/promscale> Team Lead See what we're working on (feedback welcome!): tsdb.co/prom-design-doc On Tuesday, March 16, 2021 at 7:17:07 PM UTC-4 [email protected] wrote: > Also consider this: at 30s run intervals, if you were to look at a longer > time (say a week), would you still be interested in each individual run's > timing? How would you aggregate them? > > Prometheus' answer is to construct time series of the number of runs, and > cumulative run time, starting at some arbitrary point in time (together > these are a summary). By looking at the change in these numbers over time, > we can calculate the duty cycle (what fraction of time is spent running vs. > idle) or average run time (cumulative run time divided by the number of > runs in the same timeframe). Note that this is all phrased in terms of > numbers that exist continuously (time spent since …) rather than individual > events (time spent in the fifteenth run). > > Unfortunately there is no trivial way to keep these accumulated counters > over multiple process invocations z since the client libraries only hold > them in memory. Ideally, you could get them from the long-running process > that starts these individual runs. If that is not possible, the third party > aggregating pushgateway may be useful to you. > > I hope this helps clarify how Prometheus sees the world! > > /MR > > On Wed, Mar 17, 2021, 00:06 Matthias Rampke <[email protected]> wrote: > >> There is a mismatch of models here. You are asking about plotting a set >> of (x,y) points; Prometheus fundamentally thinks in terms of continuous >> time series that happen to be sampled at the scrape interval. >> >> One way to resolve this is to consider the continuous time series of "how >> long did the last run take". This can be sampled at any time, whether 2 or >> 22 seconds after the job has finished. You need to sample (scrape) at least >> twice per job run to reliably get all counts, and you will not be directly >> able to determine whether a run happened to have exactly the same duration >> as the one before. >> >> However, you can use the knowledge of how often the job runs to make the >> graph look nice: play with the interval and overall time window alignment >> in Grafana to do that. >> >> /MR >> >> >> On Tue, Mar 16, 2021, 18:48 Dennis Kanygin <[email protected]> wrote: >> >>> Struggling with Prometheus use case. >>> >>> Periodic job runs every 30 seconds or so. Duration varies. Need a bar >>> graph with x-axis as time of run and y-axis as duration. Pushing duration >>> metrics to pushgateway. With this setup time stamp that is being attached >>> to duration metrics is the time of scrape of pushgateway. Timestamp of >>> duration is therefore "off" when graphing. As I understand, this is a >>> feature, not a bug. Any recommendation on how to accomplish what I need? >>> Seems like this would be a fairly common scenario so curious how others are >>> solving it. >>> >>> cheers, >>> >>> Dennis Kanygin >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/e69c1cb3-683a-4652-9bd4-4cd93d3b996fn%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/prometheus-users/e69c1cb3-683a-4652-9bd4-4cd93d3b996fn%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/841e4ece-7e4a-4f73-be71-43446517b178n%40googlegroups.com.
