No problem :) At least it is pretty clear for me now. I will discuss with my DPO again to have its point of view on this. Maybe I am over-reacting about those data.
Anyway, thanks for the help and the talk. It has been very interesting. Le mercredi 1 avril 2020 11:12:01 UTC+2, Stuart Clark a écrit : > > On 2020-04-01 09:58, REMI DRUILHE wrote: > > You are right, most of the time this kind de-anonymisation is extreme. > > And right again when you say that there is no personal data stored in > > Prometheus. > > > > I am also not a lawyer but I know from my DPO that the national data > > protection authority (NDPA) might be very very very meticulous, > > especially in my domain of video processing... We had several meetings > > about it and we had to review our data processing multiple time. I was > > just looking for a way to delete data with a hard deadline if the NDPA > > say that the current solution is not good enough (the one with > > storage.tsdb.retention.time option). I think it is better to come with > > an answer than saying that we did not thought about it. > > Unfortunately there are no guarantees around deletion. > > In addition to the fuzziness around exactly when a block might be > removed you can also end up with data files hanging around in certain > error scenarios (e.g. tmp files if there are issues loading the WAL on > startup or during block rotation) > > > > > Le mardi 31 mars 2020 17:51:31 UTC+2, Stuart Clark a écrit : > > > >> No that sounds fairly normal. One thing to note is that those > >> timestamps are not the times the methods were called. They are when > >> Prometheus scraped your application. So if you scrape once a minute > >> the actual call could have been at any point during that minute. > >> Equally if there are multiple calls during that minute you'd have no > >> idea when they happened either. > >> > >> I'm not a lawyer or GDPR expert, but I think the type of extreme > >> de-anonymisation you are suggesting is not generally something you'd > >> be expected to be worrying about. Equally even if you do have an > >> idea of who might have called an API there still isn't any personal > >> data in Prometheus. > >> > >> On 31 March 2020 15:27:36 BST, REMI DRUILHE <[email protected]> > >> wrote: > >> In our code, we are using a counter to count the accesses to the > >> various methods of the API. We have one counter per method. We do > >> not store the timestamp. But when we ask Prometheus with a > >> "query_range" (see request below), it returns the list of all the > >> methods that have been accessed. > >> > >> curl > >> > > ' > http://172.22.0.15:9090/api/v1/query_range?query=bea_nb_request&start=2020-03-31T00:01:00.000Z&end=2020-03-31T17:00:00.000Z&step=60s > > >> [1]' > >> > >> For each of our API method, it also returns a list of key-value > >> where the key is the timestamp and the value is the value of the > >> counter at that time (see example below). Thus, in some way, you are > >> able to track when the method has been called. And if our system is > >> used by a single user, then it is easy to follow which methods he > >> called. It is a bit twisted, but the national data protection > >> authority might also be twisted sometimes... But according to your > >> previous answers, maybe we did not used the counter in a proper way > >> and we should change the way it is designed. > >> > >> { > >> "status":"success", > >> "data":{ > >> "resultType":"matrix", > >> "result":[ > >> { > >> "metric":{ > >> "__name__":"bea_nb_request", > >> "action":"my_api_method", > >> "instance":"bea:8081", > >> "job":"bea" > >> }, > >> "values":[ > >> [ > >> 1585663440, > >> "1" > >> ], > >> [ > >> 1585663500, > >> "2" > >> ], > >> [ > >> 1585663560, > >> "3" > >> ], > >> [ > >> 1585663620, > >> "3" > >> ], > >> [ > >> 1585663680, > >> "3" > >> ], > >> [ > >> 1585663740, > >> "3" > >> ], > >> [ > >> 1585663800, > >> "3" > >> ], > >> [ > >> 1585663860, > >> "3" > >> ] > >> ] > >> }, > >> others_api_methods... > >> } > >> ] > >> } > >> } > >> > >> Le mardi 31 mars 2020 13:40:03 UTC+2, Stuart Clark a écrit : > >> How are you storing the timestamp? Is that in a label or a metric > >> value as the last call to the API? > >> > >> In general these are sounding like you are trying to store events > >> within Prometheus rather than metrics. Normally you'd not have a > >> timestamp but a counter of the number of calls to the API. > >> > >> On 31 March 2020 12:27:38 BST, REMI DRUILHE <[email protected]> > >> wrote: > >> > >> Le lundi 30 mars 2020 16:37:11 UTC+2, Brian Candler a écrit : > >> On Monday, 30 March 2020 09:34:01 UTC+1, REMI DRUILHE wrote: > >> In our context, Prometheus is storing system metrics and business > >> metrics, especially the number of accesses to the methods of our > >> API. > > > > That presumably is an aggegate of all calls to a particular method. > > > > If you recorded counts as separate metrics labelled by source IP > > address or username, then that would be identifiable. But prometheus > > does not work well with such high cardinality metrics anyway. > > > > Yeah, it is just the timestamp of the call that is stored, not the IP > > or the user name. Thus, it is not identifiable with Prometheus only. > > But, the system aims at being used by 1 or 2 persons at the same time > > in a closed network. In this context, I think it could be easy for > > someone to associate the timestamp with the person that was using the > > application at a specific time. > > > > Anyway, I will figure out another way to achieve what we would like to > > do. > > > > Thanks for the help. > > > > -- > > Sent from my Android device with K-9 Mail. Please excuse my brevity. > > > > -- > > Sent from my Android device with K-9 Mail. Please excuse my brevity. > > > > -- > > You received this message because you are subscribed to the Google > > Groups "Prometheus Users" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [email protected] <javascript:>. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/prometheus-users/29f1e59c-1d72-436a-9883-c81c71e0cbd9%40googlegroups.com > > > [2]. > > > > > > Links: > > ------ > > [1] > > > http://172.22.0.15:9090/api/v1/query_range?query=bea_nb_request&start=2020-03-31T00:01:00.000Z&end=2020-03-31T17:00:00.000Z&step=60s > > > [2] > > > https://groups.google.com/d/msgid/prometheus-users/29f1e59c-1d72-436a-9883-c81c71e0cbd9%40googlegroups.com?utm_medium=email&utm_source=footer > > > -- > Stuart Clark > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3aea1f3a-780b-43cb-81a8-8ff341a00ac2%40googlegroups.com.
