Okay I figured out what's going on, but I'm still scratching my head about how to federate the metrics gleaned by cloudwatch_exporter.
The timestamps are there for a reason -- I could ask for metrics from Cloudwatch exporter and specify "300s ago please" and cloudwatch will return "well, I've got cpu usage from this machine from 385s ago and from that machine from 445s ago..." and cloudwatch_exporter does the right thing and emits the timestamps with the metrics (if I don't tell it to omit the timestamps and prometheus assumes it's always "now"). So far so good. Now the problem I have is when I try to bring these metrics back from the small collector prometheus to the datacentre where I can afford things like 14d of local storage, Thanos, Grafana, and I have permission to emit alerts to email, Pagerduty, et al. If I ask for the latest metrics with "/federate" I get up-to-date stuff from node_exporter and process_exporter and mysqld_exporter ... but the cloudwatch_exporter stuff, because they're dated anywhere from 6 to 10 minutes ago, never appear. I tried adding --query.lookback-delta=8m but I've been warned elsewhere on this mailing list that I'm going to regret bumping it up from the default 5m... and it doesn't always work for things that are <7m old. I could tell cloudwatch_exporter to drop the timestamps on all the metrics it gleans, but then comparing ELB traffic of this exporter against request count from another exporter will always be out-of-synch (and because of that 385s...445s thing above, I can't just use a constant offset to correct it. So I'd rather avoid it if I can. How can I get a /federate scrape to bring back the latest albeit late metrics published by cloudwatch_exporter ? -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e702fd83-fa3a-4475-bd83-6a510e6aa5e8%40googlegroups.com.
