Hi, not sure if it's this what you are hitting, but the following may help:
The retention time defines how long Prometheus is expected to keep history. This does not imply that data older than this interval will vanish immediately. In other words: Once the interval has passed, Prometheus is free to delete this data, but it isn't forced to. In practice, this will depend on maintenance (compaction) running. This document provides further details, especially: https://prometheus.io/docs/prometheus/latest/storage/ "Expired block cleanup happens on a background schedule. It may take up to two hours to remove expired blocks. Expired blocks must be fully expired before they are cleaned up." Maybe this explains things for you? Kind regards, Christian On 3/26/20 3:33 PM, REMI DRUILHE wrote: > I did some others tests, on others operating systems (not sure it is > relevant because of Docker but without any clues it is better than > nothing). For anyone to test it, I also changed and reduced my > docker-compose file to: > > | > version:'3.4' > | > services: > prometheus: > image: prom/prometheus:latest > container_name: "prometheus" > depends_on: > - cadvisor > command: > - '--config.file=/etc/prometheus/prometheus.yml' > - '--storage.tsdb.retention.time=60s' > volumes: > - /tmp/prometheus:/etc/prometheus > ports: > - "9090:9090" > network_mode: "host" > cadvisor: > image: gcr.io/google-containers/cadvisor > container_name: cadvisor > volumes: > - /:/rootfs:ro > - /var/run:/var/run:ro > - /sys:/sys:ro > - /var/lib/docker/:/var/lib/docker:ro > - /dev/disk/:/dev/disk:ro > ports: > - "8080:8080" > network_mode: "host" > > And here is my prometheus.yml > > | > scrape_configs: > - job_name: cadvisor > scrape_interval: 5s > static_configs: > - targets: > - localhost:8080 > | > > But, the result is still the same with the following curl request: > > | > curl > 'http://172.21.0.3:9090/api/v1/query_range?query=go_memstats_alloc_bytes&start=2020-03-26T00:01:00.000Z&end=2020-03-26T17:00:00.000Z&step=15s' > | > > I can still get data that should have been deleted due to out of date of > the timestamp and defined in the storage.tsdb.retention.time parameter. > > According to me, it is a bug. But before opening an issue on Prometehus, > I prefer to ask for advices about it. > > Thanks, > > Best regards. > > Le mercredi 25 mars 2020 10:27:22 UTC+1, REMI DRUILHE a écrit : > > Anyone to help me on this subject? > > Thanks. > > Le mardi 24 mars 2020 13:57:05 UTC+1, REMI DRUILHE a écrit : > > Hello, > > I am using a Docker version of Prometheus (the latest one) in > which I am setting the value for the retention time (option > *storage.tsdb.retention.time*) at launch. I trying first with > small values like *1m* and *60s* (for 1 minute) or *5m* (for 5 > minutes) but I was not able to see any deletion of the data > after this period. Thus, I tested again this morning with an > higher value: *2h* (for 2 hours). I have seen this information > here > > <https://medium.com/@valyala/prometheus-storage-technical-terms-for-humans-4ab4de6c3d48> > (in the retention section) that 2h is the minimum value for > Prometheus, not sure if it is true or not. But even with this > value, the data stored 4 hours ago can still be retrieved using > the HTTP API. Here is my query: *curl > > 'http://172.18.0.3:9090/api/v1/query_range?query=go_memstats_alloc_bytes&start=2020-03-24T00:01:00.000Z&end=2020-03-24T17:00:00.000Z&step=15s > > <http://172.18.0.3:9090/api/v1/query_range?query=go_memstats_alloc_bytes&start=2020-03-24T00:01:00.000Z&end=2020-03-24T17:00:00.000Z&step=15s>'*. > > On the Prometheus GUI, the flag for this option is correctly setup. > > Here is the Docker compose file that is used to launch Prometheus: > > | > prometheus: > command:'--config.file=/etc/prometheus.yml > --storage.tsdb.retention.time=2h' > container_name:remi_prometheus > depends_on: > -cadvisor > expose: > -'9090' > image:prom/prometheus:latest > labels: > project.run.user:remi > networks: > project-bridge: > aliases: > -prometheus > ports: > -published:9090 > target:9090 > volumes: > -remi-prometheus:/prometheus:rw > - > > /home/remi/Workspace/project/runtime/configuration/prometheus.yml:/etc/prometheus.yml:rw > | > > Note that I am using cAdvisor to populate Prometheus in which I > get some Golang metrics. > > Thus here are my questions: > > * What am I doing wrong? > * Was is the minimum value for storage.tsdb.retention.time? > * Is there another option that overwrite the value set in > storage.tsdb.retention.time that I am not aware of? > * Is there a way to test that the option is working if the > minimal value is 1 day? I guess it would be to change the > system date, but not sure about it. > > > Thanks for the help, > > Best regards and good luck if you are under confinement because > of this damn virus :) > > -- > You received this message because you are subscribed to the Google > Groups "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/c3550107-62af-4120-b2d1-13926b3fff14%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/c3550107-62af-4120-b2d1-13926b3fff14%40googlegroups.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/4f712b19-5b06-5e0c-bc8f-f7624ab61634%40hoffmann-christian.info.
