Re: [prometheus-users] Re: Data retention policy

2020-04-01 Thread REMI DRUILHE 
No problem :)

At least it is pretty clear for me now. I will discuss with my DPO again to 
have its point of view on this. Maybe I am over-reacting about those data.

Anyway, thanks for the help and the talk. It has been very interesting.

Le mercredi 1 avril 2020 11:12:01 UTC+2, Stuart Clark a écrit :
>
> On 2020-04-01 09:58, REMI DRUILHE wrote: 
> > You are right, most of the time this kind de-anonymisation is extreme. 
> > And right again when you say that there is no personal data stored in 
> > Prometheus. 
> > 
> > I am also not a lawyer but I know from my DPO that the national data 
> > protection authority (NDPA) might be very very very meticulous, 
> > especially in my domain of video processing... We had several meetings 
> > about it and we had to review our data processing multiple time. I was 
> > just looking for a way to delete data with a hard deadline if the NDPA 
> > say that the current solution is not good enough (the one with 
> > storage.tsdb.retention.time option). I think it is better to come with 
> > an answer than saying that we did not thought about it. 
>
> Unfortunately there are no guarantees around deletion. 
>
> In addition to the fuzziness around exactly when a block might be 
> removed you can also end up with data files hanging around in certain 
> error scenarios (e.g. tmp files if there are issues loading the WAL on 
> startup or during block rotation) 
>
> > 
> > Le mardi 31 mars 2020 17:51:31 UTC+2, Stuart Clark a écrit : 
> > 
> >> No that sounds fairly normal. One thing to note is that those 
> >> timestamps are not the times the methods were called. They are when 
> >> Prometheus scraped your application. So if you scrape once a minute 
> >> the actual call could have been at any point during that minute. 
> >> Equally if there are multiple calls during that minute you'd have no 
> >> idea when they happened either. 
> >> 
> >> I'm not a lawyer or GDPR expert, but I think the type of extreme 
> >> de-anonymisation you are suggesting is not generally something you'd 
> >> be expected to be worrying about. Equally even if you do have an 
> >> idea of who might have called an API there still isn't any personal 
> >> data in Prometheus. 
> >> 
> >> On 31 March 2020 15:27:36 BST, REMI DRUILHE  
> >> wrote: 
> >> In our code, we are using a counter to count the accesses to the 
> >> various methods of the API. We have one counter per method. We do 
> >> not store the timestamp. But when we ask Prometheus with a 
> >> "query_range" (see request below), it returns the list of all the 
> >> methods that have been accessed. 
> >> 
> >> curl 
> >> 
> > '
> http://172.22.0.15:9090/api/v1/query_range?query=bea_nb_request=2020-03-31T00:01:00.000Z=2020-03-31T17:00:00.000Z=60s
>  
> >> [1]' 
> >> 
> >> For each of our API method, it also returns a list of key-value 
> >> where the key is the timestamp and the value is the value of the 
> >> counter at that time (see example below). Thus, in some way, you are 
> >> able to track when the method has been called. And if our system is 
> >> used by a single user, then it is easy to follow which methods he 
> >> called. It is a bit twisted, but the national data protection 
> >> authority might also be twisted sometimes... But according to your 
> >> previous answers, maybe we did not used the counter in a proper way 
> >> and we should change the way it is designed. 
> >> 
> >> { 
> >> "status":"success", 
> >> "data":{ 
> >> "resultType":"matrix", 
> >> "result":[ 
> >> { 
> >> "metric":{ 
> >> "__name__":"bea_nb_request", 
> >> "action":"my_api_method", 
> >> "instance":"bea:8081", 
> >> "job":"bea" 
> >> }, 
> >> "values":[ 
> >> [ 
> >> 1585663440, 
> >> "1" 
> >> ], 
> >> [ 
> >> 1585663500, 
> >> "2" 
> >> ], 
> >> [ 
> >> 1585663560, 
> >> "3" 
> >> ], 
> >> [ 
> >> 1585663620, 
> >> "3" 
> >> ], 
> >> [ 
> >> 1585663680, 
> >> "3" 
> >> ], 
> >> [ 
> >> 1585663740, 
> >> "3" 
> >> ], 
> >> [ 
> >> 1585663800, 
> >> "3" 
> >> ], 
> >> [ 
> >> 1585663860, 
> >> "3" 
> >> ] 
> >> ] 
> >> }, 
> >> others_api_methods... 
> >> } 
> >> ] 
> >> } 
> >> } 
> >> 
> >> Le mardi 31 mars 2020 13:40:03 UTC+2, Stuart Clark a écrit : 
> >> How are you storing the timestamp? Is that in a label or a metric 
> >> value as the last call to the API? 
> >> 
> >> In general these are sounding like you are trying to store events 
> >> within Prometheus rather than metrics. Normally you'd not have a 
> >> timestamp but a counter of the number of calls to the API. 
> >> 
> >> On 31 March 2020 12:27:38 BST, REMI DRUILHE  
> >> wrote: 
> >> 
> >> Le lundi 30 mars 2020 16:37:11 UTC+2, Brian Candler a écrit : 
> >> On Monday, 30 March 2020 09:34:01 UTC+1, REMI DRUILHE wrote: 
> >> In our context, Prometheus is storing system metrics and business 
> >> metrics, especially the number of accesses to the methods of our 
> >> API. 
> > 
> > That presumably is an aggegate of all calls to a particular method. 
> > 
> > If you recorded counts as separate

Re: [prometheus-users] Re: Data retention policy

2020-04-01 Thread Stuart Clark 

On 2020-04-01 09:58, REMI DRUILHE wrote:

You are right, most of the time this kind de-anonymisation is extreme.
And right again when you say that there is no personal data stored in
Prometheus.

I am also not a lawyer but I know from my DPO that the national data
protection authority (NDPA) might be very very very meticulous,
especially in my domain of video processing... We had several meetings
about it and we had to review our data processing multiple time. I was
just looking for a way to delete data with a hard deadline if the NDPA
say that the current solution is not good enough (the one with
storage.tsdb.retention.time option). I think it is better to come with
an answer than saying that we did not thought about it.


Unfortunately there are no guarantees around deletion.

In addition to the fuzziness around exactly when a block might be 
removed you can also end up with data files hanging around in certain 
error scenarios (e.g. tmp files if there are issues loading the WAL on 
startup or during block rotation)




Le mardi 31 mars 2020 17:51:31 UTC+2, Stuart Clark a écrit :


No that sounds fairly normal. One thing to note is that those
timestamps are not the times the methods were called. They are when
Prometheus scraped your application. So if you scrape once a minute
the actual call could have been at any point during that minute.
Equally if there are multiple calls during that minute you'd have no
idea when they happened either.

I'm not a lawyer or GDPR expert, but I think the type of extreme
de-anonymisation you are suggesting is not generally something you'd
be expected to be worrying about. Equally even if you do have an
idea of who might have called an API there still isn't any personal
data in Prometheus.

On 31 March 2020 15:27:36 BST, REMI DRUILHE 
wrote:
In our code, we are using a counter to count the accesses to the
various methods of the API. We have one counter per method. We do
not store the timestamp. But when we ask Prometheus with a
"query_range" (see request below), it returns the list of all the
methods that have been accessed.

curl


'http://172.22.0.15:9090/api/v1/query_range?query=bea_nb_request=2020-03-31T00:01:00.000Z=2020-03-31T17:00:00.000Z=60s

[1]'

For each of our API method, it also returns a list of key-value
where the key is the timestamp and the value is the value of the
counter at that time (see example below). Thus, in some way, you are
able to track when the method has been called. And if our system is
used by a single user, then it is easy to follow which methods he
called. It is a bit twisted, but the national data protection
authority might also be twisted sometimes... But according to your
previous answers, maybe we did not used the counter in a proper way
and we should change the way it is designed.

{
"status":"success",
"data":{
"resultType":"matrix",
"result":[
{
"metric":{
"__name__":"bea_nb_request",
"action":"my_api_method",
"instance":"bea:8081",
"job":"bea"
},
"values":[
[
1585663440,
"1"
],
[
1585663500,
"2"
],
[
1585663560,
"3"
],
[
1585663620,
"3"
],
[
1585663680,
"3"
],
[
1585663740,
"3"
],
[
1585663800,
"3"
],
[
1585663860,
"3"
]
]
},
others_api_methods...
}
]
}
}

Le mardi 31 mars 2020 13:40:03 UTC+2, Stuart Clark a écrit :
How are you storing the timestamp? Is that in a label or a metric
value as the last call to the API?

In general these are sounding like you are trying to store events
within Prometheus rather than metrics. Normally you'd not have a
timestamp but a counter of the number of calls to the API.

On 31 March 2020 12:27:38 BST, REMI DRUILHE 
wrote:

Le lundi 30 mars 2020 16:37:11 UTC+2, Brian Candler a écrit :
On Monday, 30 March 2020 09:34:01 UTC+1, REMI DRUILHE wrote:
In our context, Prometheus is storing system metrics and business
metrics, especially the number of accesses to the methods of our
API.


That presumably is an aggegate of all calls to a particular method.

If you recorded counts as separate metrics labelled by source IP
address or username, then that would be identifiable.  But prometheus
does not work well with such high cardinality metrics anyway.

Yeah, it is just the timestamp of the call that is stored, not the IP
or the user name. Thus, it is not identifiable with Prometheus only.
But, the system aims at being used by 1 or 2 persons at the same time
in a closed network. In this context, I think it could be easy for
someone to associate the timestamp with the person that was using the
application at a specific time.

Anyway, I will figure out another way to achieve what we would like to
do.

Thanks for the help.

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

 --
You received this message because you are subscribed to the Google
Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to prometheus-users+unsubscr...@googlegroups.com.
To view this

Re: [prometheus-users] Re: Data retention policy

2020-04-01 Thread REMI DRUILHE 
You are right, most of the time this kind de-anonymisation is extreme. And 
right again when you say that there is no personal data stored in 
Prometheus.

I am also not a lawyer but I know from my DPO that the national data 
protection authority (NDPA) might be very very very meticulous, especially 
in my domain of video processing... We had several meetings about it and we 
had to review our data processing multiple time. I was just looking for a 
way to delete data with a hard deadline if the NDPA say that the current 
solution is not good enough (the one with storage.tsdb.retention.time 
option). I think it is better to come with an answer than saying that we 
did not thought about it.

Le mardi 31 mars 2020 17:51:31 UTC+2, Stuart Clark a écrit :
>
> No that sounds fairly normal. One thing to note is that those timestamps 
> are not the times the methods were called. They are when Prometheus scraped 
> your application. So if you scrape once a minute the actual call could have 
> been at any point during that minute. Equally if there are multiple calls 
> during that minute you'd have no idea when they happened either.
>
> I'm not a lawyer or GDPR expert, but I think the type of extreme 
> de-anonymisation you are suggesting is not generally something you'd be 
> expected to be worrying about. Equally even if you do have an idea of who 
> might have called an API there still isn't any personal data in Prometheus. 
>
> On 31 March 2020 15:27:36 BST, REMI DRUILHE  > wrote:
>>
>> In our code, we are using a counter to count the accesses to the various 
>> methods of the API. We have one counter per method. We do not store the 
>> timestamp. But when we ask Prometheus with a "query_range" (see request 
>> below), it returns the list of all the methods that have been accessed. 
>>
>> curl '
>> http://172.22.0.15:9090/api/v1/query_range?query=bea_nb_request=2020-03-31T00:01:00.000Z=2020-03-31T17:00:00.000Z=60s
>> '
>>
>> For each of our API method, it also returns a list of key-value where the 
>> key is the timestamp and the value is the value of the counter at that time 
>> (see example below). Thus, in some way, you are able to track when the 
>> method has been called. And if our system is used by a single user, then it 
>> is easy to follow which methods he called. It is a bit twisted, but the 
>> national data protection authority might also be twisted sometimes... But 
>> according to your previous answers, maybe we did not used the counter in a 
>> proper way and we should change the way it is designed.
>>
>> {
>>"status":"success",
>>"data":{
>>   "resultType":"matrix",
>>   "result":[
>>  {
>> "metric":{
>>"__name__":"bea_nb_request",
>>"action":"my_api_method",
>>"instance":"bea:8081",
>>"job":"bea"
>> },
>> "values":[
>>[
>>   1585663440,
>>   "1"
>>],
>>[
>>   1585663500,
>>   "2"
>>],
>>[
>>   1585663560,
>>   "3"
>>],
>>[
>>   1585663620,
>>   "3"
>>],
>>[
>>   1585663680,
>>   "3"
>>],
>>[
>>   1585663740,
>>   "3"
>>],
>>[
>>   1585663800,
>>   "3"
>>],
>>[
>>   1585663860,
>>   "3"
>>]
>> ]
>>  },
>>  others_api_methods...
>>  }
>>   ]
>>}
>> }
>>
>>
>>
>> Le mardi 31 mars 2020 13:40:03 UTC+2, Stuart Clark a écrit :
>>>
>>> How are you storing the timestamp? Is that in a label or a metric value 
>>> as the last call to the API?
>>>
>>> In general these are sounding like you are trying to store events within 
>>> Prometheus rather than metrics. Normally you'd not have a timestamp but a 
>>> counter of the number of calls to the API. 
>>>
>>> On 31 March 2020 12:27:38 BST, REMI DRUILHE  wrote:



 Le lundi 30 mars 2020 16:37:11 UTC+2, Brian Candler a écrit :
>
> On Monday, 30 March 2020 09:34:01 UTC+1, REMI DRUILHE wrote:
>>
>> In our context, Prometheus is storing system metrics and business 
>> metrics, especially the number of accesses to the methods of our API.
>>
>>>

> That presumably is an aggegate of all calls to a particular method.
>
> If you recorded counts as separate metrics labelled by source IP 
> address or username, then that would be identifiable.  But prometheus 
> does 
> not work well with such high cardinality metrics anyway.
>

 Yeah, it is just the timestamp of the call that is stored, not the IP 
 or the user

Re: [prometheus-users] Re: Data retention policy

2020-03-31 Thread Stuart Clark 
No that sounds fairly normal. One thing to note is that those timestamps are 
not the times the methods were called. They are when Prometheus scraped your 
application. So if you scrape once a minute the actual call could have been at 
any point during that minute. Equally if there are multiple calls during that 
minute you'd have no idea when they happened either.

I'm not a lawyer or GDPR expert, but I think the type of extreme 
de-anonymisation you are suggesting is not generally something you'd be 
expected to be worrying about. Equally even if you do have an idea of who might 
have called an API there still isn't any personal data in Prometheus. 

On 31 March 2020 15:27:36 BST, REMI DRUILHE  wrote:
>In our code, we are using a counter to count the accesses to the
>various 
>methods of the API. We have one counter per method. We do not store the
>
>timestamp. But when we ask Prometheus with a "query_range" (see request
>
>below), it returns the list of all the methods that have been accessed.
>
>
>curl 
>'http://172.22.0.15:9090/api/v1/query_range?query=bea_nb_request=2020-03-31T00:01:00.000Z=2020-03-31T17:00:00.000Z=60s'
>
>For each of our API method, it also returns a list of key-value where
>the 
>key is the timestamp and the value is the value of the counter at that
>time 
>(see example below). Thus, in some way, you are able to track when the 
>method has been called. And if our system is used by a single user,
>then it 
>is easy to follow which methods he called. It is a bit twisted, but the
>
>national data protection authority might also be twisted sometimes...
>But 
>according to your previous answers, maybe we did not used the counter
>in a 
>proper way and we should change the way it is designed.
>
>{
>   "status":"success",
>   "data":{
>  "resultType":"matrix",
>  "result":[
> {
>"metric":{
>   "__name__":"bea_nb_request",
>   "action":"my_api_method",
>   "instance":"bea:8081",
>   "job":"bea"
>},
>"values":[
>   [
>  1585663440,
>  "1"
>   ],
>   [
>  1585663500,
>  "2"
>   ],
>   [
>  1585663560,
>  "3"
>   ],
>   [
>  1585663620,
>  "3"
>   ],
>   [
>  1585663680,
>  "3"
>   ],
>   [
>  1585663740,
>  "3"
>   ],
>   [
>  1585663800,
>  "3"
>   ],
>   [
>  1585663860,
>  "3"
>   ]
>]
> },
> others_api_methods...
> }
>  ]
>   }
>}
>
>
>
>Le mardi 31 mars 2020 13:40:03 UTC+2, Stuart Clark a écrit :
>>
>> How are you storing the timestamp? Is that in a label or a metric
>value as 
>> the last call to the API?
>>
>> In general these are sounding like you are trying to store events
>within 
>> Prometheus rather than metrics. Normally you'd not have a timestamp
>but a 
>> counter of the number of calls to the API. 
>>
>> On 31 March 2020 12:27:38 BST, REMI DRUILHE > > wrote:
>>>
>>>
>>>
>>> Le lundi 30 mars 2020 16:37:11 UTC+2, Brian Candler a écrit :

 On Monday, 30 March 2020 09:34:01 UTC+1, REMI DRUILHE wrote:
>
> In our context, Prometheus is storing system metrics and business 
> metrics, especially the number of accesses to the methods of our
>API.
>
>>
>>>
 That presumably is an aggegate of all calls to a particular method.

 If you recorded counts as separate metrics labelled by source IP
>address 
 or username, then that would be identifiable.  But prometheus does
>not work 
 well with such high cardinality metrics anyway.

>>>
>>> Yeah, it is just the timestamp of the call that is stored, not the
>IP or 
>>> the user name. Thus, it is not identifiable with Prometheus only.
>But, the 
>>> system aims at being used by 1 or 2 persons at the same time in a
>closed 
>>> network. In this context, I think it could be easy for someone to
>associate 
>>> the timestamp with the person that was using the application at a
>specific 
>>> time.
>>>
>>> Anyway, I will figure out another way to achieve what we would like
>to do.
>>>
>>> Thanks for the help.
>>>
>>>
>> -- 
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>
>
>-- 
>You received this message because you are subscribed to the Google
>Groups "Prometheus Users" group.
>To unsubscribe from this group and stop receiving emails from it, send
>an email to prometheus-users+unsubscr...@googlegroups.com.
>To view this discussion on the web visit
>https://groups.google.com/d/msgid/prometheus-users/67393d18-2c1c-4a79-a780-365111fc40e6%40googlegroups.com.

-- 
Sent from my Android device

Re: [prometheus-users] Re: Data retention policy

2020-03-31 Thread REMI DRUILHE 
In our code, we are using a counter to count the accesses to the various 
methods of the API. We have one counter per method. We do not store the 
timestamp. But when we ask Prometheus with a "query_range" (see request 
below), it returns the list of all the methods that have been accessed. 

curl 
'http://172.22.0.15:9090/api/v1/query_range?query=bea_nb_request=2020-03-31T00:01:00.000Z=2020-03-31T17:00:00.000Z=60s'

For each of our API method, it also returns a list of key-value where the 
key is the timestamp and the value is the value of the counter at that time 
(see example below). Thus, in some way, you are able to track when the 
method has been called. And if our system is used by a single user, then it 
is easy to follow which methods he called. It is a bit twisted, but the 
national data protection authority might also be twisted sometimes... But 
according to your previous answers, maybe we did not used the counter in a 
proper way and we should change the way it is designed.

{
   "status":"success",
   "data":{
  "resultType":"matrix",
  "result":[
 {
"metric":{
   "__name__":"bea_nb_request",
   "action":"my_api_method",
   "instance":"bea:8081",
   "job":"bea"
},
"values":[
   [
  1585663440,
  "1"
   ],
   [
  1585663500,
  "2"
   ],
   [
  1585663560,
  "3"
   ],
   [
  1585663620,
  "3"
   ],
   [
  1585663680,
  "3"
   ],
   [
  1585663740,
  "3"
   ],
   [
  1585663800,
  "3"
   ],
   [
  1585663860,
  "3"
   ]
]
 },
 others_api_methods...
 }
  ]
   }
}



Le mardi 31 mars 2020 13:40:03 UTC+2, Stuart Clark a écrit :
>
> How are you storing the timestamp? Is that in a label or a metric value as 
> the last call to the API?
>
> In general these are sounding like you are trying to store events within 
> Prometheus rather than metrics. Normally you'd not have a timestamp but a 
> counter of the number of calls to the API. 
>
> On 31 March 2020 12:27:38 BST, REMI DRUILHE  > wrote:
>>
>>
>>
>> Le lundi 30 mars 2020 16:37:11 UTC+2, Brian Candler a écrit :
>>>
>>> On Monday, 30 March 2020 09:34:01 UTC+1, REMI DRUILHE wrote:

 In our context, Prometheus is storing system metrics and business 
 metrics, especially the number of accesses to the methods of our API.

>
>>
>>> That presumably is an aggegate of all calls to a particular method.
>>>
>>> If you recorded counts as separate metrics labelled by source IP address 
>>> or username, then that would be identifiable.  But prometheus does not work 
>>> well with such high cardinality metrics anyway.
>>>
>>
>> Yeah, it is just the timestamp of the call that is stored, not the IP or 
>> the user name. Thus, it is not identifiable with Prometheus only. But, the 
>> system aims at being used by 1 or 2 persons at the same time in a closed 
>> network. In this context, I think it could be easy for someone to associate 
>> the timestamp with the person that was using the application at a specific 
>> time.
>>
>> Anyway, I will figure out another way to achieve what we would like to do.
>>
>> Thanks for the help.
>>
>>
> -- 
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/67393d18-2c1c-4a79-a780-365111fc40e6%40googlegroups.com.

Re: [prometheus-users] Re: Data retention policy

2020-03-31 Thread Stuart Clark 
How are you storing the timestamp? Is that in a label or a metric value as the 
last call to the API?

In general these are sounding like you are trying to store events within 
Prometheus rather than metrics. Normally you'd not have a timestamp but a 
counter of the number of calls to the API. 

On 31 March 2020 12:27:38 BST, REMI DRUILHE  wrote:
>
>
>Le lundi 30 mars 2020 16:37:11 UTC+2, Brian Candler a écrit :
>>
>> On Monday, 30 March 2020 09:34:01 UTC+1, REMI DRUILHE wrote:
>>>
>>> In our context, Prometheus is storing system metrics and business 
>>> metrics, especially the number of accesses to the methods of our
>API.
>>>

>
>> That presumably is an aggegate of all calls to a particular method.
>>
>> If you recorded counts as separate metrics labelled by source IP
>address 
>> or username, then that would be identifiable.  But prometheus does
>not work 
>> well with such high cardinality metrics anyway.
>>
>
>Yeah, it is just the timestamp of the call that is stored, not the IP
>or 
>the user name. Thus, it is not identifiable with Prometheus only. But,
>the 
>system aims at being used by 1 or 2 persons at the same time in a
>closed 
>network. In this context, I think it could be easy for someone to
>associate 
>the timestamp with the person that was using the application at a
>specific 
>time.
>
>Anyway, I will figure out another way to achieve what we would like to
>do.
>
>Thanks for the help.
>
>-- 
>You received this message because you are subscribed to the Google
>Groups "Prometheus Users" group.
>To unsubscribe from this group and stop receiving emails from it, send
>an email to prometheus-users+unsubscr...@googlegroups.com.
>To view this discussion on the web visit
>https://groups.google.com/d/msgid/prometheus-users/40f42a0c-64ef-4ec4-93f8-2ef0cf65a6c8%40googlegroups.com.

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/D580F35A-7514-4CF0-9382-3022AEE1772C%40Jahingo.com.

Re: [prometheus-users] Re: Data retention policy

2020-03-31 Thread REMI DRUILHE 


Le lundi 30 mars 2020 16:37:11 UTC+2, Brian Candler a écrit :
>
> On Monday, 30 March 2020 09:34:01 UTC+1, REMI DRUILHE wrote:
>>
>> In our context, Prometheus is storing system metrics and business 
>> metrics, especially the number of accesses to the methods of our API.
>>
>>>

> That presumably is an aggegate of all calls to a particular method.
>
> If you recorded counts as separate metrics labelled by source IP address 
> or username, then that would be identifiable.  But prometheus does not work 
> well with such high cardinality metrics anyway.
>

Yeah, it is just the timestamp of the call that is stored, not the IP or 
the user name. Thus, it is not identifiable with Prometheus only. But, the 
system aims at being used by 1 or 2 persons at the same time in a closed 
network. In this context, I think it could be easy for someone to associate 
the timestamp with the person that was using the application at a specific 
time.

Anyway, I will figure out another way to achieve what we would like to do.

Thanks for the help.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/40f42a0c-64ef-4ec4-93f8-2ef0cf65a6c8%40googlegroups.com.

Re: [prometheus-users] Re: Data retention policy

2020-03-30 Thread Brian Candler 
On Monday, 30 March 2020 09:34:01 UTC+1, REMI DRUILHE wrote:
>
> In our context, Prometheus is storing system metrics and business metrics, 
> especially the number of accesses to the methods of our API.
>
>>
>>>
That presumably is an aggegate of all calls to a particular method.

If you recorded counts as separate metrics labelled by source IP address or 
username, then that would be identifiable.  But prometheus does not work 
well with such high cardinality metrics anyway.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/6d60bad3-db8a-4452-adf1-b88bd4fd947b%40googlegroups.com.

Re: [prometheus-users] Re: Data retention policy

2020-03-30 Thread REMI DRUILHE 
In our context, Prometheus is storing system metrics and business metrics, 
especially the number of accesses to the methods of our API. We do not have 
any direct user data stored in Prometheus. But I think that someone could 
cross-reference the data to find something more specific about someone (it 
is a big if) that is why I wanted to have a hard deadline for the deletion 
of the data.

Le vendredi 27 mars 2020 18:39:27 UTC+1, Ben Kochie a écrit :
>
> Yes, this is the general guideline. User identifiable data should not be 
> in Prometheus. Mostly because it tends to be a cardinality problem.
>
> On Fri, Mar 27, 2020, 14:58 Stuart Clark  > wrote:
>
>> On 27/03/2020 13:55, REMI DRUILHE wrote:
>>
>> Hi, 
>>
>> Thanks for the answer.
>>
>> Just to add some context to what I want to do. We are trying to make our 
>> application (Prometheus being a part of it) compliant with GDPR. The 
>> default period for our data retention is 10 days but can be set by the user 
>> to 1h minimum. If the data are not deleted after 1 hour then, it is no 
>> longer compliant with what we declared to the national data protection 
>> authority.
>>
>> From your answer, I understand now that the parameter 
>> *storage.tsdb.retention.time* is not hard deadline for the data. Data 
>> might live after the retention time, at least 2 hours. So I launched a new 
>> test with storage.tsdb.retention.time equals to 5m. Then, I waited for 3 
>> hours to see if the data were deleted but they are always in Prometheus. 
>>
>> Is it possible to anticipate when the data will be deleted to update ours 
>> compliance with GDPR? Can we change the background schedule to have a 
>> better estimation of when the data will be deleted? Or should I use the 
>> delete method of Prometheus form an external component to be sure that the 
>> data are deleted on time?
>>
>>
>> What data are you expecting Prometheus to hold that would be subject to 
>> GDPR?
>>
>> Prometheus stores monitoring metrics so wouldn't have anything related to 
>> personal data, so isn't in scope for GDPR...
>>
>>
>> Thanks,
>>
>> Best regards.
>>
>>
>> Le jeudi 26 mars 2020 21:40:35 UTC+1, Christian Hoffmann a écrit : 
>>>
>>> Hi, 
>>>
>>> not sure if it's this what you are hitting, but the following may help: 
>>>
>>> The retention time defines how long Prometheus is expected to keep 
>>> history. This does not imply that data older than this interval will 
>>> vanish immediately. 
>>> In other words: Once the interval has passed, Prometheus is free to 
>>> delete this data, but it isn't forced to. In practice, this will depend 
>>> on maintenance (compaction) running. 
>>>
>>> This document provides further details, especially: 
>>>
>>> https://prometheus.io/docs/prometheus/latest/storage/ 
>>> "Expired block cleanup happens on a background schedule. It may take up 
>>> to two hours to remove expired blocks. Expired blocks must be fully 
>>> expired before they are cleaned up." 
>>>
>>> Maybe this explains things for you? 
>>>
>>> Kind regards, 
>>> Christian 
>>>
>>> On 3/26/20 3:33 PM, REMI DRUILHE wrote: 
>>> > I did some others tests, on others operating systems (not sure it is 
>>> > relevant because of Docker but without any clues it is better than 
>>> > nothing). For anyone to test it, I also changed and reduced my 
>>> > docker-compose file to: 
>>> > 
>>> > | 
>>> > version:'3.4' 
>>> > | 
>>> > services: 
>>> >   prometheus: 
>>> > image: prom/prometheus:latest 
>>> > container_name: "prometheus" 
>>> > depends_on: 
>>> >   - cadvisor 
>>> > command: 
>>> >   - '--config.file=/etc/prometheus/prometheus.yml' 
>>> >   - '--storage.tsdb.retention.time=60s' 
>>> > volumes: 
>>> >   - /tmp/prometheus:/etc/prometheus 
>>> > ports: 
>>> >   - "9090:9090" 
>>> > network_mode: "host" 
>>> >   cadvisor: 
>>> > image: gcr.io/google-containers/cadvisor 
>>> > container_name: cadvisor 
>>> > volumes: 
>>> >   - /:/rootfs:ro 
>>> >   - /var/run:/var/run:ro 
>>> >   - /sys:/sys:ro 
>>> >   - /var/lib/docker/:/var/lib/docker:ro 
>>> >   - /dev/disk/:/dev/disk:ro 
>>> > ports: 
>>> >   - "8080:8080" 
>>> > network_mode: "host" 
>>> > 
>>> > And here is my prometheus.yml 
>>> > 
>>> > | 
>>> > scrape_configs: 
>>> > - job_name: cadvisor 
>>> >   scrape_interval: 5s 
>>> >   static_configs: 
>>> >   - targets: 
>>> > - localhost:8080 
>>> > | 
>>> > 
>>> > But, the result is still the same with the following curl request:  
>>> > 
>>> > | 
>>> > curl 
>>> > '
>>> http://172.21.0.3:9090/api/v1/query_range?query=go_memstats_alloc_bytes=2020-03-26T00:01:00.000Z=2020-03-26T17:00:00.000Z=15s'
>>>  
>>>
>>> > | 
>>> > 
>>> > I can still get data that should have been deleted due to out of date 
>>> of 
>>> > the timestamp and defined in the storage.tsdb.retention.time 
>>> parameter. 
>>> > 
>>> > According to me, it is a bug. But before opening an issue on 
>>> Prometehus, 
>>> > I prefer to ask

Re: [prometheus-users] Re: Data retention policy

2020-03-27 Thread Ben Kochie 
Yes, this is the general guideline. User identifiable data should not be in
Prometheus. Mostly because it tends to be a cardinality problem.

On Fri, Mar 27, 2020, 14:58 Stuart Clark  wrote:

> On 27/03/2020 13:55, REMI DRUILHE wrote:
>
> Hi,
>
> Thanks for the answer.
>
> Just to add some context to what I want to do. We are trying to make our
> application (Prometheus being a part of it) compliant with GDPR. The
> default period for our data retention is 10 days but can be set by the user
> to 1h minimum. If the data are not deleted after 1 hour then, it is no
> longer compliant with what we declared to the national data protection
> authority.
>
> From your answer, I understand now that the parameter
> *storage.tsdb.retention.time* is not hard deadline for the data. Data
> might live after the retention time, at least 2 hours. So I launched a new
> test with storage.tsdb.retention.time equals to 5m. Then, I waited for 3
> hours to see if the data were deleted but they are always in Prometheus.
>
> Is it possible to anticipate when the data will be deleted to update ours
> compliance with GDPR? Can we change the background schedule to have a
> better estimation of when the data will be deleted? Or should I use the
> delete method of Prometheus form an external component to be sure that the
> data are deleted on time?
>
>
> What data are you expecting Prometheus to hold that would be subject to
> GDPR?
>
> Prometheus stores monitoring metrics so wouldn't have anything related to
> personal data, so isn't in scope for GDPR...
>
>
> Thanks,
>
> Best regards.
>
>
> Le jeudi 26 mars 2020 21:40:35 UTC+1, Christian Hoffmann a écrit :
>>
>> Hi,
>>
>> not sure if it's this what you are hitting, but the following may help:
>>
>> The retention time defines how long Prometheus is expected to keep
>> history. This does not imply that data older than this interval will
>> vanish immediately.
>> In other words: Once the interval has passed, Prometheus is free to
>> delete this data, but it isn't forced to. In practice, this will depend
>> on maintenance (compaction) running.
>>
>> This document provides further details, especially:
>>
>> https://prometheus.io/docs/prometheus/latest/storage/
>> "Expired block cleanup happens on a background schedule. It may take up
>> to two hours to remove expired blocks. Expired blocks must be fully
>> expired before they are cleaned up."
>>
>> Maybe this explains things for you?
>>
>> Kind regards,
>> Christian
>>
>> On 3/26/20 3:33 PM, REMI DRUILHE wrote:
>> > I did some others tests, on others operating systems (not sure it is
>> > relevant because of Docker but without any clues it is better than
>> > nothing). For anyone to test it, I also changed and reduced my
>> > docker-compose file to:
>> >
>> > |
>> > version:'3.4'
>> > |
>> > services:
>> >   prometheus:
>> > image: prom/prometheus:latest
>> > container_name: "prometheus"
>> > depends_on:
>> >   - cadvisor
>> > command:
>> >   - '--config.file=/etc/prometheus/prometheus.yml'
>> >   - '--storage.tsdb.retention.time=60s'
>> > volumes:
>> >   - /tmp/prometheus:/etc/prometheus
>> > ports:
>> >   - "9090:9090"
>> > network_mode: "host"
>> >   cadvisor:
>> > image: gcr.io/google-containers/cadvisor
>> > container_name: cadvisor
>> > volumes:
>> >   - /:/rootfs:ro
>> >   - /var/run:/var/run:ro
>> >   - /sys:/sys:ro
>> >   - /var/lib/docker/:/var/lib/docker:ro
>> >   - /dev/disk/:/dev/disk:ro
>> > ports:
>> >   - "8080:8080"
>> > network_mode: "host"
>> >
>> > And here is my prometheus.yml
>> >
>> > |
>> > scrape_configs:
>> > - job_name: cadvisor
>> >   scrape_interval: 5s
>> >   static_configs:
>> >   - targets:
>> > - localhost:8080
>> > |
>> >
>> > But, the result is still the same with the following curl request:
>> >
>> > |
>> > curl
>> > '
>> http://172.21.0.3:9090/api/v1/query_range?query=go_memstats_alloc_bytes=2020-03-26T00:01:00.000Z=2020-03-26T17:00:00.000Z=15s'
>>
>> > |
>> >
>> > I can still get data that should have been deleted due to out of date
>> of
>> > the timestamp and defined in the storage.tsdb.retention.time parameter.
>> >
>> > According to me, it is a bug. But before opening an issue on
>> Prometehus,
>> > I prefer to ask for advices about it.
>> >
>> > Thanks,
>> >
>> > Best regards.
>> >
>> > Le mercredi 25 mars 2020 10:27:22 UTC+1, REMI DRUILHE a écrit :
>> >
>> > Anyone to help me on this subject?
>> >
>> > Thanks.
>> >
>> > Le mardi 24 mars 2020 13:57:05 UTC+1, REMI DRUILHE a écrit :
>> >
>> > Hello,
>> >
>> > I am using a Docker version of Prometheus (the latest one) in
>> > which I am setting the value for the retention time (option
>> > *storage.tsdb.retention.time*) at launch. I trying first with
>> > small values like *1m* and *60s* (for 1 minute) or *5m* (for 5
>> > minutes) but I was not able to see any deletion of the data
>> >

Re: [prometheus-users] Re: Data retention policy

2020-03-27 Thread Stuart Clark 

On 27/03/2020 13:55, REMI DRUILHE wrote:

Hi,

Thanks for the answer.

Just to add some context to what I want to do. We are trying to make 
our application (Prometheus being a part of it) compliant with GDPR. 
The default period for our data retention is 10 days but can be set by 
the user to 1h minimum. If the data are not deleted after 1 hour then, 
it is no longer compliant with what we declared to the national data 
protection authority.


From your answer, I understand now that the parameter 
*storage.tsdb.retention.time* is not hard deadline for the data. Data 
might live after the retention time, at least 2 hours. So I launched a 
new test with storage.tsdb.retention.time equals to 5m. Then, I waited 
for 3 hours to see if the data were deleted but they are always in 
Prometheus.


Is it possible to anticipate when the data will be deleted to update 
ours compliance with GDPR? Can we change the background schedule to 
have a better estimation of when the data will be deleted? Or should I 
use the delete method of Prometheus form an external component to be 
sure that the data are deleted on time?




What data are you expecting Prometheus to hold that would be subject to 
GDPR?


Prometheus stores monitoring metrics so wouldn't have anything related 
to personal data, so isn't in scope for GDPR...




Thanks,

Best regards.


Le jeudi 26 mars 2020 21:40:35 UTC+1, Christian Hoffmann a écrit :

Hi,

not sure if it's this what you are hitting, but the following may
help:

The retention time defines how long Prometheus is expected to keep
history. This does not imply that data older than this interval will
vanish immediately.
In other words: Once the interval has passed, Prometheus is free to
delete this data, but it isn't forced to. In practice, this will
depend
on maintenance (compaction) running.

This document provides further details, especially:

https://prometheus.io/docs/prometheus/latest/storage/

"Expired block cleanup happens on a background schedule. It may
take up
to two hours to remove expired blocks. Expired blocks must be fully
expired before they are cleaned up."

Maybe this explains things for you?

Kind regards,
Christian

On 3/26/20 3:33 PM, REMI DRUILHE wrote:
> I did some others tests, on others operating systems (not sure
it is
> relevant because of Docker but without any clues it is better than
> nothing). For anyone to test it, I also changed and reduced my
> docker-compose file to:
>
> |
> version:'3.4'
> |
> services:
>   prometheus:
>     image: prom/prometheus:latest
>     container_name: "prometheus"
>     depends_on:
>       - cadvisor
>     command:
>       - '--config.file=/etc/prometheus/prometheus.yml'
>       - '--storage.tsdb.retention.time=60s'
>     volumes:
>       - /tmp/prometheus:/etc/prometheus
>     ports:
>       - "9090:9090"
>     network_mode: "host"
>   cadvisor:
>     image: gcr.io/google-containers/cadvisor

>     container_name: cadvisor
>     volumes:
>       - /:/rootfs:ro
>       - /var/run:/var/run:ro
>       - /sys:/sys:ro
>       - /var/lib/docker/:/var/lib/docker:ro
>       - /dev/disk/:/dev/disk:ro
>     ports:
>       - "8080:8080"
>     network_mode: "host"
>
> And here is my prometheus.yml
>
> |
> scrape_configs:
> - job_name: cadvisor
>   scrape_interval: 5s
>   static_configs:
>   - targets:
>     - localhost:8080
> |
>
> But, the result is still the same with the following curl request:
>
> |
> curl
>

'http://172.21.0.3:9090/api/v1/query_range?query=go_memstats_alloc_bytes=2020-03-26T00:01:00.000Z=2020-03-26T17:00:00.000Z=15s

'

> |
>
> I can still get data that should have been deleted due to out of
date of
> the timestamp and defined in the storage.tsdb.retention.time
parameter.
>
> According to me, it is a bug. But before opening an issue on
Prometehus,
> I prefer to ask for advices about it.
>
> Thanks,
>
> Best regards.
>
> Le mercredi 25 mars 2020 10:27:22 UTC+1, REMI DRUILHE a écrit :
>
>     Anyone to help me on this subject?
>
>     Thanks.
>
>     Le mardi 24 mars 2020 13:57:05 UTC+1, REMI DRUILHE a écrit :
>
>         Hello,
>
>         I am using a Docker version of Prometheus (the latest
one) in
>         which I am setting the value for the retention time (option
>         *storage.tsdb.retention.time*) at launch. I trying first
with
>         small values like *1m* and *60s*

Re: [prometheus-users] Re: Data retention policy

2020-03-27 Thread REMI DRUILHE 
Hi,

Thanks for the answer.

Just to add some context to what I want to do. We are trying to make our 
application (Prometheus being a part of it) compliant with GDPR. The 
default period for our data retention is 10 days but can be set by the user 
to 1h minimum. If the data are not deleted after 1 hour then, it is no 
longer compliant with what we declared to the national data protection 
authority.

>From your answer, I understand now that the parameter 
*storage.tsdb.retention.time* is not hard deadline for the data. Data might 
live after the retention time, at least 2 hours. So I launched a new test 
with storage.tsdb.retention.time equals to 5m. Then, I waited for 3 hours 
to see if the data were deleted but they are always in Prometheus. 

Is it possible to anticipate when the data will be deleted to update ours 
compliance with GDPR? Can we change the background schedule to have a 
better estimation of when the data will be deleted? Or should I use the 
delete method of Prometheus form an external component to be sure that the 
data are deleted on time?

Thanks,

Best regards.


Le jeudi 26 mars 2020 21:40:35 UTC+1, Christian Hoffmann a écrit :
>
> Hi, 
>
> not sure if it's this what you are hitting, but the following may help: 
>
> The retention time defines how long Prometheus is expected to keep 
> history. This does not imply that data older than this interval will 
> vanish immediately. 
> In other words: Once the interval has passed, Prometheus is free to 
> delete this data, but it isn't forced to. In practice, this will depend 
> on maintenance (compaction) running. 
>
> This document provides further details, especially: 
>
> https://prometheus.io/docs/prometheus/latest/storage/ 
> "Expired block cleanup happens on a background schedule. It may take up 
> to two hours to remove expired blocks. Expired blocks must be fully 
> expired before they are cleaned up." 
>
> Maybe this explains things for you? 
>
> Kind regards, 
> Christian 
>
> On 3/26/20 3:33 PM, REMI DRUILHE wrote: 
> > I did some others tests, on others operating systems (not sure it is 
> > relevant because of Docker but without any clues it is better than 
> > nothing). For anyone to test it, I also changed and reduced my 
> > docker-compose file to: 
> > 
> > | 
> > version:'3.4' 
> > | 
> > services: 
> >   prometheus: 
> > image: prom/prometheus:latest 
> > container_name: "prometheus" 
> > depends_on: 
> >   - cadvisor 
> > command: 
> >   - '--config.file=/etc/prometheus/prometheus.yml' 
> >   - '--storage.tsdb.retention.time=60s' 
> > volumes: 
> >   - /tmp/prometheus:/etc/prometheus 
> > ports: 
> >   - "9090:9090" 
> > network_mode: "host" 
> >   cadvisor: 
> > image: gcr.io/google-containers/cadvisor 
> > container_name: cadvisor 
> > volumes: 
> >   - /:/rootfs:ro 
> >   - /var/run:/var/run:ro 
> >   - /sys:/sys:ro 
> >   - /var/lib/docker/:/var/lib/docker:ro 
> >   - /dev/disk/:/dev/disk:ro 
> > ports: 
> >   - "8080:8080" 
> > network_mode: "host" 
> > 
> > And here is my prometheus.yml 
> > 
> > | 
> > scrape_configs: 
> > - job_name: cadvisor 
> >   scrape_interval: 5s 
> >   static_configs: 
> >   - targets: 
> > - localhost:8080 
> > | 
> > 
> > But, the result is still the same with the following curl request:  
> > 
> > | 
> > curl 
> > '
> http://172.21.0.3:9090/api/v1/query_range?query=go_memstats_alloc_bytes=2020-03-26T00:01:00.000Z=2020-03-26T17:00:00.000Z=15s'
>  
>
> > | 
> > 
> > I can still get data that should have been deleted due to out of date of 
> > the timestamp and defined in the storage.tsdb.retention.time parameter. 
> > 
> > According to me, it is a bug. But before opening an issue on Prometehus, 
> > I prefer to ask for advices about it. 
> > 
> > Thanks, 
> > 
> > Best regards. 
> > 
> > Le mercredi 25 mars 2020 10:27:22 UTC+1, REMI DRUILHE a écrit : 
> > 
> > Anyone to help me on this subject? 
> > 
> > Thanks. 
> > 
> > Le mardi 24 mars 2020 13:57:05 UTC+1, REMI DRUILHE a écrit : 
> > 
> > Hello, 
> > 
> > I am using a Docker version of Prometheus (the latest one) in 
> > which I am setting the value for the retention time (option 
> > *storage.tsdb.retention.time*) at launch. I trying first with 
> > small values like *1m* and *60s* (for 1 minute) or *5m* (for 5 
> > minutes) but I was not able to see any deletion of the data 
> > after this period. Thus, I tested again this morning with an 
> > higher value: *2h* (for 2 hours). I have seen this information 
> > here 
> > <
> https://medium.com/@valyala/prometheus-storage-technical-terms-for-humans-4ab4de6c3d48>
>  
>
> > (in the retention section) that 2h is the minimum value for 
> > Prometheus, not sure if it is true or not. But even with this 
> > value, the data stored 4 hours ago can still be retrieved using 
> >

Re: [prometheus-users] Re: Data retention policy

2020-03-26 Thread Christian Hoffmann 
Hi,

not sure if it's this what you are hitting, but the following may help:

The retention time defines how long Prometheus is expected to keep
history. This does not imply that data older than this interval will
vanish immediately.
In other words: Once the interval has passed, Prometheus is free to
delete this data, but it isn't forced to. In practice, this will depend
on maintenance (compaction) running.

This document provides further details, especially:

https://prometheus.io/docs/prometheus/latest/storage/
"Expired block cleanup happens on a background schedule. It may take up
to two hours to remove expired blocks. Expired blocks must be fully
expired before they are cleaned up."

Maybe this explains things for you?

Kind regards,
Christian

On 3/26/20 3:33 PM, REMI DRUILHE wrote:
> I did some others tests, on others operating systems (not sure it is
> relevant because of Docker but without any clues it is better than
> nothing). For anyone to test it, I also changed and reduced my
> docker-compose file to:
> 
> |
> version:'3.4'
> |
> services:
>   prometheus:
>     image: prom/prometheus:latest
>     container_name: "prometheus"
>     depends_on:
>       - cadvisor
>     command:
>       - '--config.file=/etc/prometheus/prometheus.yml'
>       - '--storage.tsdb.retention.time=60s'
>     volumes:
>       - /tmp/prometheus:/etc/prometheus
>     ports:
>       - "9090:9090"
>     network_mode: "host"
>   cadvisor:
>     image: gcr.io/google-containers/cadvisor
>     container_name: cadvisor
>     volumes:
>       - /:/rootfs:ro
>       - /var/run:/var/run:ro
>       - /sys:/sys:ro
>       - /var/lib/docker/:/var/lib/docker:ro
>       - /dev/disk/:/dev/disk:ro
>     ports:
>       - "8080:8080"
>     network_mode: "host"
> 
> And here is my prometheus.yml
> 
> |
> scrape_configs:
> - job_name: cadvisor
>   scrape_interval: 5s
>   static_configs:
>   - targets:
>     - localhost:8080
> |
> 
> But, the result is still the same with the following curl request: 
> 
> |
> curl
> 'http://172.21.0.3:9090/api/v1/query_range?query=go_memstats_alloc_bytes=2020-03-26T00:01:00.000Z=2020-03-26T17:00:00.000Z=15s'
> |
> 
> I can still get data that should have been deleted due to out of date of
> the timestamp and defined in the storage.tsdb.retention.time parameter.
> 
> According to me, it is a bug. But before opening an issue on Prometehus,
> I prefer to ask for advices about it.
> 
> Thanks,
> 
> Best regards.
> 
> Le mercredi 25 mars 2020 10:27:22 UTC+1, REMI DRUILHE a écrit :
> 
> Anyone to help me on this subject?
> 
> Thanks.
> 
> Le mardi 24 mars 2020 13:57:05 UTC+1, REMI DRUILHE a écrit :
> 
> Hello,
> 
> I am using a Docker version of Prometheus (the latest one) in
> which I am setting the value for the retention time (option
> *storage.tsdb.retention.time*) at launch. I trying first with
> small values like *1m* and *60s* (for 1 minute) or *5m* (for 5
> minutes) but I was not able to see any deletion of the data
> after this period. Thus, I tested again this morning with an
> higher value: *2h* (for 2 hours). I have seen this information
> here
> 
> 
> (in the retention section) that 2h is the minimum value for
> Prometheus, not sure if it is true or not. But even with this
> value, the data stored 4 hours ago can still be retrieved using
> the HTTP API. Here is my query: *curl
> 
> 'http://172.18.0.3:9090/api/v1/query_range?query=go_memstats_alloc_bytes=2020-03-24T00:01:00.000Z=2020-03-24T17:00:00.000Z=15s
> 
> '*.
> 
> On the Prometheus GUI, the flag for this option is correctly setup.
> 
> Here is the Docker compose file that is used to launch Prometheus:
> 
> |
>   prometheus:
>     command:'--config.file=/etc/prometheus.yml
> --storage.tsdb.retention.time=2h'
>     container_name:remi_prometheus
>     depends_on:
>     -cadvisor
>     expose:
>     -'9090'
>     image:prom/prometheus:latest
>     labels:
>       project.run.user:remi
>     networks:
>       project-bridge:
>         aliases:
>         -prometheus
>     ports:
>     -published:9090
>       target:9090
>     volumes:
>     -remi-prometheus:/prometheus:rw
>     -
> 
> /home/remi/Workspace/project/runtime/configuration/prometheus.yml:/etc/prometheus.yml:rw
> |
> 
> Note that I am using cAdvisor to populate Prometheus in which I
> get some Golang metrics.
> 
> Thus here are my questions:
> 
>   * What am I doing wrong?
>   * Was is the minimum