Hi, I'm fairly new to Prometheus so bear with me. What I'm trying to do is graph the number of EC2 Instances that have been deleted by a tool of mine. I'm not even sure Prometheus is the right tool for the job.
I have a tool that's persistently running. Every hour or so it terminates a number of unused AWS EC2 Instances and exports the numbers at a /metrics endpoint. It currently generates a GaugeMetricFamily called cleaned_instances_total with the labels [cloud, region, type] containing e.g. ['aws', 'us-west-2', 'm5.xlarge'] and as value the number of Instances it just removed. Prometheus scrapes the target like twice an hour. Now the first Problem I ran into is I'd like to graph the number of instances that were removed in a day, per region and type. However because Prometheus scrapes the target more often than the instances are being cleaned, I get repeating values. Very visible on this output: [image: cleaned_instances.png] Like with my human eye I can tell that those exact repeating values are likely because the metrics were scraped before the next cleanup run occurred. But I don't know how to express that in a PromQL query. I suppose I could make the Gauge a Counter, but even then; lets say I have two metrics where 10 instances each have been terminated, how would I know if the 10 instances in my second timestamp are the same 10 from the first one, or if the tool terminated 10 instances, was restarted and the Counter reset to 0 and then terminated another 10 instances? I guess one question right now would be, is there a way to deduplicate those identical values? Like, if all labels and values at a point in time are the same as the ones from previous points in time consider it as a single timestamp... or something along those lines. Is Prometheus even the right tool for what I'm trying to do? Basically I'm not trying to graph something that happens over time, I'm trying to graph some number of events that happen at a point in time. I'm just very confused right now with all of Prometheus' capabilities and also can't quite wrap my head around the problem. Any help is much appreciated. Thanks, -- Lukas -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/bca2396c-f917-4a6f-840a-5cad52f96d81%40googlegroups.com.
