Re: [prometheus-developers] Multi-Tenant ACLs with Prometheus
I built this based on prom-label-proxy to support mapping basicAuth to a set of tenant label matchers https://github.com/kfdm/promql-guard On Sat, Feb 15, 2020 at 1:43 AM Simon Pasquier wrote: > > You can have a look at those 2 projects too: > https://github.com/hoffie/prometheus-filter-proxy > https://github.com/openshift/prom-label-proxy > > > On Fri, Feb 14, 2020 at 4:13 PM Conrad Wood wrote: > > > > On Fri, 2020-02-14 at 15:58 +0100, Julius Volz wrote: > > > On Fri, Feb 14, 2020 at 2:01 PM Conrad Wood > > > wrote: > > > > That is a good point, I did not consider that labels might have > > > > been > > > > aggregated away. Clearly that needs to be considered. > > > > > > > > However, the ACLEvaluator also needs to look at the label values > > > > once > > > > the query returns(see Example 2). Or is there a way that one can > > > > query > > > > > > Note that putting IP addresses or similar high-cardinality items into > > > Prometheus label names doesn't work well unless the possible set of > > > values is restricted to a reasonable number. Otherwise you'll blow up > > > your Prometheus server immediately (a big server can do a couple > > > million series that are present at the same time, and every unique > > > combination of label values creates one series, so usually just > > > putting public IPs into label values is a no-starter, since it > > > multiplies up with other labels very quickly). > > > > > > > for all values of a set of labels for a given metric without the > > > > datapoints in a given timerange? > > > > > > There is: > > > https://prometheus.io/docs/prometheus/latest/querying/api/#querying-metadata > > > > > > ...but again, you'll probably run into cardinality overload if you > > > have an unbounded number of IPs in label values. > > > > > > > Also, the "parsing of promql" - is that available in a library or > > > > as an > > > > RPC? If not, would that be also considered out-of-scope? > > > > > > You would use Prometheus's "promql" Go package: > > > https://godoc.org/github.com/prometheus/prometheus/promql > > > > > > > > > Thank you, > > > > I hear you re unbound labels, especially with IPs. In this case there > > is quite a small set of IPs (<2048), probably even less. > > > > The information you send is most helpful. I believe I now have > > sufficient information to fix something up that will address my usecase > > outside of prometheus. > > > > Thanks again, > > > > Conrad > > > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Prometheus Developers" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to prometheus-developers+unsubscr...@googlegroups.com. > > To view this discussion on the web visit > > https://groups.google.com/d/msgid/prometheus-developers/b556929c59a02d04b92ad6c9422e1ae7fca8c78b.camel%40conradwood.net. > > > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to prometheus-developers+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-developers/CAM6RFu4nFa4yy5PqFUPgTSYKoazDQG44F2vOJJi9wAp3rnv%3DDw%40mail.gmail.com. -- Paul Traylor http://kungfudiscomonkey.net/ -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAOOkOjjTik5ZU%2BmZS34iczhVwfB9%2Bh-B-FTgfuq5kEW45QRftg%40mail.gmail.com.
Re: [prometheus-developers] Multi-Tenant ACLs with Prometheus
You can have a look at those 2 projects too: https://github.com/hoffie/prometheus-filter-proxy https://github.com/openshift/prom-label-proxy On Fri, Feb 14, 2020 at 4:13 PM Conrad Wood wrote: > > On Fri, 2020-02-14 at 15:58 +0100, Julius Volz wrote: > > On Fri, Feb 14, 2020 at 2:01 PM Conrad Wood > > wrote: > > > That is a good point, I did not consider that labels might have > > > been > > > aggregated away. Clearly that needs to be considered. > > > > > > However, the ACLEvaluator also needs to look at the label values > > > once > > > the query returns(see Example 2). Or is there a way that one can > > > query > > > > Note that putting IP addresses or similar high-cardinality items into > > Prometheus label names doesn't work well unless the possible set of > > values is restricted to a reasonable number. Otherwise you'll blow up > > your Prometheus server immediately (a big server can do a couple > > million series that are present at the same time, and every unique > > combination of label values creates one series, so usually just > > putting public IPs into label values is a no-starter, since it > > multiplies up with other labels very quickly). > > > > > for all values of a set of labels for a given metric without the > > > datapoints in a given timerange? > > > > There is: > > https://prometheus.io/docs/prometheus/latest/querying/api/#querying-metadata > > > > ...but again, you'll probably run into cardinality overload if you > > have an unbounded number of IPs in label values. > > > > > Also, the "parsing of promql" - is that available in a library or > > > as an > > > RPC? If not, would that be also considered out-of-scope? > > > > You would use Prometheus's "promql" Go package: > > https://godoc.org/github.com/prometheus/prometheus/promql > > > > > Thank you, > > I hear you re unbound labels, especially with IPs. In this case there > is quite a small set of IPs (<2048), probably even less. > > The information you send is most helpful. I believe I now have > sufficient information to fix something up that will address my usecase > outside of prometheus. > > Thanks again, > > Conrad > > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to prometheus-developers+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-developers/b556929c59a02d04b92ad6c9422e1ae7fca8c78b.camel%40conradwood.net. > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAM6RFu4nFa4yy5PqFUPgTSYKoazDQG44F2vOJJi9wAp3rnv%3DDw%40mail.gmail.com.
Re: [prometheus-developers] Multi-Tenant ACLs with Prometheus
On Fri, 2020-02-14 at 15:58 +0100, Julius Volz wrote: > On Fri, Feb 14, 2020 at 2:01 PM Conrad Wood > wrote: > > That is a good point, I did not consider that labels might have > > been > > aggregated away. Clearly that needs to be considered. > > > > However, the ACLEvaluator also needs to look at the label values > > once > > the query returns(see Example 2). Or is there a way that one can > > query > > Note that putting IP addresses or similar high-cardinality items into > Prometheus label names doesn't work well unless the possible set of > values is restricted to a reasonable number. Otherwise you'll blow up > your Prometheus server immediately (a big server can do a couple > million series that are present at the same time, and every unique > combination of label values creates one series, so usually just > putting public IPs into label values is a no-starter, since it > multiplies up with other labels very quickly). > > > for all values of a set of labels for a given metric without the > > datapoints in a given timerange? > > There is: > https://prometheus.io/docs/prometheus/latest/querying/api/#querying-metadata > > ...but again, you'll probably run into cardinality overload if you > have an unbounded number of IPs in label values. > > > Also, the "parsing of promql" - is that available in a library or > > as an > > RPC? If not, would that be also considered out-of-scope? > > You would use Prometheus's "promql" Go package: > https://godoc.org/github.com/prometheus/prometheus/promql > Thank you, I hear you re unbound labels, especially with IPs. In this case there is quite a small set of IPs (<2048), probably even less. The information you send is most helpful. I believe I now have sufficient information to fix something up that will address my usecase outside of prometheus. Thanks again, Conrad -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/b556929c59a02d04b92ad6c9422e1ae7fca8c78b.camel%40conradwood.net.
Re: [prometheus-developers] Multi-Tenant ACLs with Prometheus
On Fri, Feb 14, 2020 at 2:01 PM Conrad Wood wrote: > That is a good point, I did not consider that labels might have been > aggregated away. Clearly that needs to be considered. > > However, the ACLEvaluator also needs to look at the label values once > the query returns(see Example 2). Or is there a way that one can query > Note that putting IP addresses or similar high-cardinality items into Prometheus label names doesn't work well unless the possible set of values is restricted to a reasonable number. Otherwise you'll blow up your Prometheus server immediately (a big server can do a couple million series that are present at the same time, and every unique combination of label values creates one series, so usually just putting public IPs into label values is a no-starter, since it multiplies up with other labels very quickly). > for all values of a set of labels for a given metric without the > datapoints in a given timerange? > There is: https://prometheus.io/docs/prometheus/latest/querying/api/#querying-metadata ...but again, you'll probably run into cardinality overload if you have an unbounded number of IPs in label values. Also, the "parsing of promql" - is that available in a library or as an > RPC? If not, would that be also considered out-of-scope? > You would use Prometheus's "promql" Go package: https://godoc.org/github.com/prometheus/prometheus/promql Conrad > > > > On Tue, 2020-02-11 at 15:59 +0100, Julius Volz wrote: > > Just to add to this, I would unfortunately agree that this has always > > been considered as out of scope for Prometheus, whereas Cortex > > explicitly was designed around multi-tenancy, and could be a good > > option here. > > > > Regarding this piece of your proposal doc (just to keep the > > discussion here): > > > > > > 1. Call ACLEvaluator with the result of the query > > 2. Filter out results which are not granted > > 3. Return all remaining results > > 4. Return to the caller > > > > > > Note that this would not be correct. The result of a query might have > > already aggregated away or changed labels that you might need to use > > for evaluating ACLs. You'd need to validate queries at the point > > where they load series data, which currently is the two AST nodes > > VectorSelector and MatrixSelector. That's what a gateway in front of > > Prometheus would have to do: > > > > 1. Parse the PromQL query > > 2. Find any vector / matrix selector nodes > > 3. Verify that a tenant label is provided in them (or add one) > > 4. Forward the query > > > > On Tue, Feb 11, 2020 at 12:02 PM Conrad Wood > > wrote: > > > On Tue, 2020-02-11 at 09:55 +, Brian Brazil wrote: > > > > On Tue, 11 Feb 2020 at 09:48, Conrad Wood > > > wrote: > > > > > On Tue, 2020-02-11 at 09:31 +, Brian Brazil wrote: > > > > > > On Tue, 11 Feb 2020 at 09:13, Conrad Wood > > > > > > > > wrote: > > > > > > > On Tue, 2020-02-11 at 08:51 +, Brian Brazil wrote: > > > > > > > > On Tue, 11 Feb 2020 at 08:41, Conrad Wood < > > > c...@conradwood.net > > > > > > > > > > > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > I have some thoughts on how prometheus can help to > > > support > > > > > > > multi- > > > > > > > > > tenant set ups (without going down the rabbit hole of > > > > > > > > > authentication/authorisation). > > > > > > > > > I frequently find myself in a position where I want to > > > > > expose > > > > > > > data > > > > > > > > > in prometheus to external parties. These parties are > > > > > > > commercially > > > > > > > > > independent of each other, so I need to be specific > > > which > > > > > users > > > > > > > may > > > > > > > > > access which data. > > > > > > > > > Ideally, I like prometheus to intercept queries for > > > data > > > > > not > > > > > > > > > belonging to a given user. > > > > > > > > > That would allow me (and others) to expose prometheus' > > > API > > > > > > > through > > > > > > > > > an authenticating reverse proxy to users (and run > > > grafanas > > > > > for > > > > > > > > > users or they can run it themselves even). > > > > > > > > > > > > > > > > > > A more detailed proposal is in this google doc: Multi- > > > > > Tenant > > > > > > > ACLs > > > > > > > > > with Prometheus > > > > > > > > > > > > > > > > > > Would a patch along those lines be something the > > > prometheus > > > > > > > > > development team would accept? > > > > > > > > > If not, I would kindly request some comments what ought > > > to > > > > > be > > > > > > > > > addressed in order for it to become acceptable. I think > > > > > this > > > > > > > would > > > > > > > > > be useful feature for quite a few parties. > > > > > > > > > > > > > > > > > > > > > > > > > This is out of scope for Prometheus itself, among other > > > > > things > > > > > > > promql > > > > > > > > evaluation should never have a hard dependency on the > > > > > network. > > > > > > > > > > > > > > > > However there are already solutions in this space such > > > as >
