On Fri, Feb 14, 2020 at 2:01 PM Conrad Wood <[email protected]> wrote:
> That is a good point, I did not consider that labels might have been > aggregated away. Clearly that needs to be considered. > > However, the ACLEvaluator also needs to look at the label values once > the query returns(see Example 2). Or is there a way that one can query > Note that putting IP addresses or similar high-cardinality items into Prometheus label names doesn't work well unless the possible set of values is restricted to a reasonable number. Otherwise you'll blow up your Prometheus server immediately (a big server can do a couple million series that are present at the same time, and every unique combination of label values creates one series, so usually just putting public IPs into label values is a no-starter, since it multiplies up with other labels very quickly). > for all values of a set of labels for a given metric without the > datapoints in a given timerange? > There is: https://prometheus.io/docs/prometheus/latest/querying/api/#querying-metadata ...but again, you'll probably run into cardinality overload if you have an unbounded number of IPs in label values. Also, the "parsing of promql" - is that available in a library or as an > RPC? If not, would that be also considered out-of-scope? > You would use Prometheus's "promql" Go package: https://godoc.org/github.com/prometheus/prometheus/promql Conrad > > > > On Tue, 2020-02-11 at 15:59 +0100, Julius Volz wrote: > > Just to add to this, I would unfortunately agree that this has always > > been considered as out of scope for Prometheus, whereas Cortex > > explicitly was designed around multi-tenancy, and could be a good > > option here. > > > > Regarding this piece of your proposal doc (just to keep the > > discussion here): > > > > ---- > > 1. Call ACLEvaluator with the result of the query > > 2. Filter out results which are not granted > > 3. Return all remaining results > > 4. Return to the caller > > ---- > > > > Note that this would not be correct. The result of a query might have > > already aggregated away or changed labels that you might need to use > > for evaluating ACLs. You'd need to validate queries at the point > > where they load series data, which currently is the two AST nodes > > VectorSelector and MatrixSelector. That's what a gateway in front of > > Prometheus would have to do: > > > > 1. Parse the PromQL query > > 2. Find any vector / matrix selector nodes > > 3. Verify that a tenant label is provided in them (or add one) > > 4. Forward the query > > > > On Tue, Feb 11, 2020 at 12:02 PM Conrad Wood <[email protected]> > > wrote: > > > On Tue, 2020-02-11 at 09:55 +0000, Brian Brazil wrote: > > > > On Tue, 11 Feb 2020 at 09:48, Conrad Wood <[email protected]> > > > wrote: > > > > > On Tue, 2020-02-11 at 09:31 +0000, Brian Brazil wrote: > > > > > > On Tue, 11 Feb 2020 at 09:13, Conrad Wood <[email protected] > > > > > > > > > wrote: > > > > > > > On Tue, 2020-02-11 at 08:51 +0000, Brian Brazil wrote: > > > > > > > > On Tue, 11 Feb 2020 at 08:41, Conrad Wood < > > > [email protected] > > > > > > > > > > > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > I have some thoughts on how prometheus can help to > > > support > > > > > > > multi- > > > > > > > > > tenant set ups (without going down the rabbit hole of > > > > > > > > > authentication/authorisation). > > > > > > > > > I frequently find myself in a position where I want to > > > > > expose > > > > > > > data > > > > > > > > > in prometheus to external parties. These parties are > > > > > > > commercially > > > > > > > > > independent of each other, so I need to be specific > > > which > > > > > users > > > > > > > may > > > > > > > > > access which data. > > > > > > > > > Ideally, I like prometheus to intercept queries for > > > data > > > > > not > > > > > > > > > belonging to a given user. > > > > > > > > > That would allow me (and others) to expose prometheus' > > > API > > > > > > > through > > > > > > > > > an authenticating reverse proxy to users (and run > > > grafanas > > > > > for > > > > > > > > > users or they can run it themselves even). > > > > > > > > > > > > > > > > > > A more detailed proposal is in this google doc: Multi- > > > > > Tenant > > > > > > > ACLs > > > > > > > > > with Prometheus > > > > > > > > > > > > > > > > > > Would a patch along those lines be something the > > > prometheus > > > > > > > > > development team would accept? > > > > > > > > > If not, I would kindly request some comments what ought > > > to > > > > > be > > > > > > > > > addressed in order for it to become acceptable. I think > > > > > this > > > > > > > would > > > > > > > > > be useful feature for quite a few parties. > > > > > > > > > > > > > > > > > > > > > > > > > This is out of scope for Prometheus itself, among other > > > > > things > > > > > > > promql > > > > > > > > evaluation should never have a hard dependency on the > > > > > network. > > > > > > > > > > > > > > > > However there are already solutions in this space such > > > as > > > > > > > > https://github.com/weeco/cortex-gateway/ and > > > > > > > > https://github.com/rancher/prometheus-auth that do this > > > > > entirely > > > > > > > via > > > > > > > > a reverse proxy. > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > thank you for your swift response. > > > > > > > I fully understand the desire to keep it out of prometheus > > > and > > > > > move > > > > > > > this in front of prometheus, like prometheus-auth does. > > > > > > > > > > > > > > AFAICT this requires a duplication of the PromQL parser > > > code, > > > > > which > > > > > > > I > > > > > > > consider troubling from a security perspective. If > > > prometheus' > > > > > > > promql > > > > > > > and that of the proxy diverge, it may be possible that data > > > is > > > > > > > being > > > > > > > leaked between parties. It is also quite hard to verify > > > > > > > correctness. > > > > > > > > > > > > Selectors aren't that hard to spot, and you can always choose > > > to > > > > > fail > > > > > > closed. > > > > > > > > > > > > > In the usecase I described, IMHO the issue of PromQL > > > network > > > > > access > > > > > > > is > > > > > > > perhaps less relevant, given that (parts of) the network > > > must > > > > > work > > > > > > > for > > > > > > > the user to authenticate (ldap) and access the api (http) > > > and > > > > > the > > > > > > > reverse proxy too. The disks may well be on a NAS too. > > > > > > > > > > > > Using a NAS for Prometheus storage would not be recommended > > > for > > > > > > reliability reasons. > > > > > > > > > > > > > In my proposal, the PromQL rule evaluation, etc. would not > > > > > require > > > > > > > any > > > > > > > more network access than currently. *Only* Api calls. > > > > > > > > > > > > > > I would definitely consider it good practice to install the > > > > > > > ACLEvaluator on the same machine as prometheus and deploy > > > > > caching > > > > > > > strategies within it. > > > > > > > > > > > > > > Might it be preferable to use a local socket interface > > > instead > > > > > of > > > > > > > gRPC > > > > > > > to mitigate the issue of network access? > > > > > > > > > > > > That's still network access. As long as I can get to a > > > running > > > > > > Prometheus, I should be able to successfully execute PromQL - > > > no > > > > > > matter what else is broken network wise. > > > > > > > > > > > > This is best done entirely in a reverse proxy. > > > > > > > > > > I see your point re "most succesfully execute PromQL", that is > > > > > indeed > > > > > rather important for a monitoring system that might aid in > > > finding > > > > > out > > > > > why something is borked. > > > > > > > > > > In the case of remote_read, surely that is a bit relaxed. Would > > > it > > > > > thus > > > > > be an option to pass the received headers&cookies to the > > > > > remote_read > > > > > backend? That way the backend could do network magic if it > > > needs > > > > > to, > > > > > but the localstorage would still satisfy above requirement. > > > > > > > > That's getting a bit magic. A remote storage's results should > > > only > > > > really depend on the query itself, otherwise it'd be interesting > > > to > > > > debug. > > > > The standard way to approach this is by injecting a label. > > > > > > Thank you for your time. I understand that there is little to no > > > chance > > > of getting such a feature merged. > > > I will pursue alternate routes. > > > > > > Thank you. > > > > > > Conrad > > > > > > > > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CA%2BT6YoyHaEPA1rnSYfRzrfF3Nxt33LxgAdtVqWy8yyTO2wFjow%40mail.gmail.com.
