On 10/09/2020 10:48, Martin Emrich wrote:
Hello!
Our prometheus contains metrics of our kubernetes cluster.
Today, I wanted to look at data of a node (EC2 instance on AWS) that
has been removed yesterday. But all the data regarding the VM is gone.
In contrast, data of a Kubernetes Pod that has been terminated is
still visible later.
The nodes are discovered using ec2_sd_configs, the Kuberneted Pod data
is scraped via federation from another prometheus instance running
inside the K8s cluster.
Apparently there is a difference in behaviour regarding removed
targets. But I cannot find a configuration parameter.
How can I keep historic data of removed VMs (or other targets) when
the target itself is no longer there? I bet that is a FAQ, but I did
not find information on how data removal works (beyond the storage
configuration, which I expect works the same for all series)
How are you trying to see the older data & what is your retention period?
If a pod/EC2 instance goes away it just means no new data will be
scraped and any timeseries that were being scraped will go stale, but
the existing data should remain until the retention period is reached.
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/64deecd3-6f03-d884-2d85-e5cc95904524%40Jahingo.com.
