Thanks Brian. You are correct. We are using kube-state-metrics to export. The behavior of the exporter is to export unless and until the object is deleted from the cluster. As we were tracking jobs in this case, they are not cleaned up automatically. Hence the exporter exports the metric even after the job finishes execution.
On Mon, Apr 27, 2020 at 12:14 PM Brian Candler <[email protected]> wrote: > Try plotting graphs separately of: > > kube_job_status_completion_time{namespace="some-namespace", job_name=~ > "some-job-\\d+"} > > kube_job_status_start_time{namespace="some-namespace", job_name=~ > "some-job-\\d+"} > > and I think it will become clearer. > > Each metric has values at different times, from when the metric was first > exported until it stopped being exported. In this case each metric has a > constant value, but it still exists for a range of times. > > A query like "A - B", taken at instant t, returns a result whenever both > metrics (with matching labels) have a value at that time. > > If you give such a query to grafana, then it will sweep t over a range, > which is the selected time range for the graph, and the graph will show a > point whenever the expression has a value. > > I am guessing that your exporter was returning values for both > kube_job_status_completion_time and kube_job_status_start_time (for that > particular job) between 4:43am and 6:06am, and therefore Grafana will show > the value of the expression when both metrics have a value. > > For example: maybe kube_job_status_start_time existed from 4:40am to > 06:06am, and kube_job_status_completion_time existed from 4:43am to > 06:10am. Your graph will show values for all the times that both metrics > exist, i.e. 4:43am to 6:06am. > >> -- > You received this message because you are subscribed to a topic in the > Google Groups "Prometheus Users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/prometheus-users/HTkKApeIJmE/unsubscribe > . > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/101a37d2-03c9-480c-88f1-03a0a0ea6048%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/101a37d2-03c9-480c-88f1-03a0a0ea6048%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAMn2_1ijKg6vhhTe9tvNDaXWibn5ykEY%2B5SX%3D_-Xe0G_be%2BALw%40mail.gmail.com.
