Hi, Thanks for the answer.
Just to add some context to what I want to do. We are trying to make our application (Prometheus being a part of it) compliant with GDPR. The default period for our data retention is 10 days but can be set by the user to 1h minimum. If the data are not deleted after 1 hour then, it is no longer compliant with what we declared to the national data protection authority. >From your answer, I understand now that the parameter *storage.tsdb.retention.time* is not hard deadline for the data. Data might live after the retention time, at least 2 hours. So I launched a new test with storage.tsdb.retention.time equals to 5m. Then, I waited for 3 hours to see if the data were deleted but they are always in Prometheus. Is it possible to anticipate when the data will be deleted to update ours compliance with GDPR? Can we change the background schedule to have a better estimation of when the data will be deleted? Or should I use the delete method of Prometheus form an external component to be sure that the data are deleted on time? Thanks, Best regards. Le jeudi 26 mars 2020 21:40:35 UTC+1, Christian Hoffmann a écrit : > > Hi, > > not sure if it's this what you are hitting, but the following may help: > > The retention time defines how long Prometheus is expected to keep > history. This does not imply that data older than this interval will > vanish immediately. > In other words: Once the interval has passed, Prometheus is free to > delete this data, but it isn't forced to. In practice, this will depend > on maintenance (compaction) running. > > This document provides further details, especially: > > https://prometheus.io/docs/prometheus/latest/storage/ > "Expired block cleanup happens on a background schedule. It may take up > to two hours to remove expired blocks. Expired blocks must be fully > expired before they are cleaned up." > > Maybe this explains things for you? > > Kind regards, > Christian > > On 3/26/20 3:33 PM, REMI DRUILHE wrote: > > I did some others tests, on others operating systems (not sure it is > > relevant because of Docker but without any clues it is better than > > nothing). For anyone to test it, I also changed and reduced my > > docker-compose file to: > > > > | > > version:'3.4' > > | > > services: > > prometheus: > > image: prom/prometheus:latest > > container_name: "prometheus" > > depends_on: > > - cadvisor > > command: > > - '--config.file=/etc/prometheus/prometheus.yml' > > - '--storage.tsdb.retention.time=60s' > > volumes: > > - /tmp/prometheus:/etc/prometheus > > ports: > > - "9090:9090" > > network_mode: "host" > > cadvisor: > > image: gcr.io/google-containers/cadvisor > > container_name: cadvisor > > volumes: > > - /:/rootfs:ro > > - /var/run:/var/run:ro > > - /sys:/sys:ro > > - /var/lib/docker/:/var/lib/docker:ro > > - /dev/disk/:/dev/disk:ro > > ports: > > - "8080:8080" > > network_mode: "host" > > > > And here is my prometheus.yml > > > > | > > scrape_configs: > > - job_name: cadvisor > > scrape_interval: 5s > > static_configs: > > - targets: > > - localhost:8080 > > | > > > > But, the result is still the same with the following curl request: > > > > | > > curl > > ' > http://172.21.0.3:9090/api/v1/query_range?query=go_memstats_alloc_bytes&start=2020-03-26T00:01:00.000Z&end=2020-03-26T17:00:00.000Z&step=15s' > > > > | > > > > I can still get data that should have been deleted due to out of date of > > the timestamp and defined in the storage.tsdb.retention.time parameter. > > > > According to me, it is a bug. But before opening an issue on Prometehus, > > I prefer to ask for advices about it. > > > > Thanks, > > > > Best regards. > > > > Le mercredi 25 mars 2020 10:27:22 UTC+1, REMI DRUILHE a écrit : > > > > Anyone to help me on this subject? > > > > Thanks. > > > > Le mardi 24 mars 2020 13:57:05 UTC+1, REMI DRUILHE a écrit : > > > > Hello, > > > > I am using a Docker version of Prometheus (the latest one) in > > which I am setting the value for the retention time (option > > *storage.tsdb.retention.time*) at launch. I trying first with > > small values like *1m* and *60s* (for 1 minute) or *5m* (for 5 > > minutes) but I was not able to see any deletion of the data > > after this period. Thus, I tested again this morning with an > > higher value: *2h* (for 2 hours). I have seen this information > > here > > < > https://medium.com/@valyala/prometheus-storage-technical-terms-for-humans-4ab4de6c3d48> > > > > (in the retention section) that 2h is the minimum value for > > Prometheus, not sure if it is true or not. But even with this > > value, the data stored 4 hours ago can still be retrieved using > > the HTTP API. Here is my query: *curl > > ' > http://172.18.0.3:9090/api/v1/query_range?query=go_memstats_alloc_bytes&start=2020-03-24T00:01:00.000Z&end=2020-03-24T17:00:00.000Z&step=15s > > > < > http://172.18.0.3:9090/api/v1/query_range?query=go_memstats_alloc_bytes&start=2020-03-24T00:01:00.000Z&end=2020-03-24T17:00:00.000Z&step=15s>'*. > > > > > > On the Prometheus GUI, the flag for this option is correctly > setup. > > > > Here is the Docker compose file that is used to launch > Prometheus: > > > > | > > prometheus: > > command:'--config.file=/etc/prometheus.yml > > --storage.tsdb.retention.time=2h' > > container_name:remi_prometheus > > depends_on: > > -cadvisor > > expose: > > -'9090' > > image:prom/prometheus:latest > > labels: > > project.run.user:remi > > networks: > > project-bridge: > > aliases: > > -prometheus > > ports: > > -published:9090 > > target:9090 > > volumes: > > -remi-prometheus:/prometheus:rw > > - > > > /home/remi/Workspace/project/runtime/configuration/prometheus.yml:/etc/prometheus.yml:rw > > > > | > > > > Note that I am using cAdvisor to populate Prometheus in which I > > get some Golang metrics. > > > > Thus here are my questions: > > > > * What am I doing wrong? > > * Was is the minimum value for storage.tsdb.retention.time? > > * Is there another option that overwrite the value set in > > storage.tsdb.retention.time that I am not aware of? > > * Is there a way to test that the option is working if the > > minimal value is 1 day? I guess it would be to change the > > system date, but not sure about it. > > > > > > Thanks for the help, > > > > Best regards and good luck if you are under confinement because > > of this damn virus :) > > > > -- > > You received this message because you are subscribed to the Google > > Groups "Prometheus Users" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to promethe...@googlegroups.com <javascript:> > > <mailto:prometheus-users+unsubscr...@googlegroups.com <javascript:>>. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/prometheus-users/c3550107-62af-4120-b2d1-13926b3fff14%40googlegroups.com > > > < > https://groups.google.com/d/msgid/prometheus-users/c3550107-62af-4120-b2d1-13926b3fff14%40googlegroups.com?utm_medium=email&utm_source=footer>. > > > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/27f3c76c-f2b3-4bfe-87d7-02adbf61f8c4%40googlegroups.com.
