Re: [prometheus-users] Re: Data retention policy

2020-04-01 Thread REMI DRUILHE
No problem :) At least it is pretty clear for me now. I will discuss with my DPO again to have its point of view on this. Maybe I am over-reacting about those data. Anyway, thanks for the help and the talk. It has been very interesting. Le mercredi 1 avril 2020 11:12:01 UTC+2, Stuart Clark a

Re: [prometheus-users] Re: Data retention policy

2020-04-01 Thread Stuart Clark
On 2020-04-01 09:58, REMI DRUILHE wrote: You are right, most of the time this kind de-anonymisation is extreme. And right again when you say that there is no personal data stored in Prometheus. I am also not a lawyer but I know from my DPO that the national data protection authority (NDPA)

Re: [prometheus-users] Re: Data retention policy

2020-04-01 Thread REMI DRUILHE
You are right, most of the time this kind de-anonymisation is extreme. And right again when you say that there is no personal data stored in Prometheus. I am also not a lawyer but I know from my DPO that the national data protection authority (NDPA) might be very very very meticulous,

Re: [prometheus-users] Re: Data retention policy

2020-03-31 Thread Stuart Clark
No that sounds fairly normal. One thing to note is that those timestamps are not the times the methods were called. They are when Prometheus scraped your application. So if you scrape once a minute the actual call could have been at any point during that minute. Equally if there are multiple

Re: [prometheus-users] Re: Data retention policy

2020-03-31 Thread REMI DRUILHE
In our code, we are using a counter to count the accesses to the various methods of the API. We have one counter per method. We do not store the timestamp. But when we ask Prometheus with a "query_range" (see request below), it returns the list of all the methods that have been accessed. curl

Re: [prometheus-users] Re: Data retention policy

2020-03-31 Thread Stuart Clark
How are you storing the timestamp? Is that in a label or a metric value as the last call to the API? In general these are sounding like you are trying to store events within Prometheus rather than metrics. Normally you'd not have a timestamp but a counter of the number of calls to the API.

Re: [prometheus-users] Re: Data retention policy

2020-03-31 Thread REMI DRUILHE
Le lundi 30 mars 2020 16:37:11 UTC+2, Brian Candler a écrit : > > On Monday, 30 March 2020 09:34:01 UTC+1, REMI DRUILHE wrote: >> >> In our context, Prometheus is storing system metrics and business >> metrics, especially the number of accesses to the methods of our API. >> >>> > That

Re: [prometheus-users] Re: Data retention policy

2020-03-30 Thread Brian Candler
On Monday, 30 March 2020 09:34:01 UTC+1, REMI DRUILHE wrote: > > In our context, Prometheus is storing system metrics and business metrics, > especially the number of accesses to the methods of our API. > >> >>> That presumably is an aggegate of all calls to a particular method. If you recorded

Re: [prometheus-users] Re: Data retention policy

2020-03-30 Thread REMI DRUILHE
In our context, Prometheus is storing system metrics and business metrics, especially the number of accesses to the methods of our API. We do not have any direct user data stored in Prometheus. But I think that someone could cross-reference the data to find something more specific about someone

Re: [prometheus-users] Re: Data retention policy

2020-03-27 Thread Ben Kochie
Yes, this is the general guideline. User identifiable data should not be in Prometheus. Mostly because it tends to be a cardinality problem. On Fri, Mar 27, 2020, 14:58 Stuart Clark wrote: > On 27/03/2020 13:55, REMI DRUILHE wrote: > > Hi, > > Thanks for the answer. > > Just to add some context

Re: [prometheus-users] Re: Data retention policy

2020-03-27 Thread Stuart Clark
On 27/03/2020 13:55, REMI DRUILHE wrote: Hi, Thanks for the answer. Just to add some context to what I want to do. We are trying to make our application (Prometheus being a part of it) compliant with GDPR. The default period for our data retention is 10 days but can be set by the user to 1h

Re: [prometheus-users] Re: Data retention policy

2020-03-27 Thread REMI DRUILHE
Hi, Thanks for the answer. Just to add some context to what I want to do. We are trying to make our application (Prometheus being a part of it) compliant with GDPR. The default period for our data retention is 10 days but can be set by the user to 1h minimum. If the data are not deleted after

Re: [prometheus-users] Re: Data retention policy

2020-03-26 Thread Christian Hoffmann
Hi, not sure if it's this what you are hitting, but the following may help: The retention time defines how long Prometheus is expected to keep history. This does not imply that data older than this interval will vanish immediately. In other words: Once the interval has passed, Prometheus is free