> The good thing with it is that you could have easily multiple secret > backends configured to retrieve secrets for specific "service" (so > that you > could keep "generic airflow's secerts" in one backend but still have > possibility of custom operators to use other backends (with different > authentication, scopes etc.).
Having the ability to configure multiple secrets backends is independent of this feature. The original PR/AIP to add Secrets Backends decided to leave this ability out as it was more complex to configure. We could add that back in. I still don't quite get from your example where you are proposing this would be used? Can you give a fuller example please? Do you have a concrete use case where you need this? Not everything in Airflow needs to be a hook; just access the secrets backend directly. I'm not sure what wrapping an extra layer around these classes gives us? Without a concrete example I can't see anything other than this adds a lot of complexity. -ash On May 18 2020, at 2:45 pm, Jarek Potiuk <jarek.pot...@polidea.com> wrote: > Hello Everyone, > > TL;DR; I was just about to start to work on a small set of Hooks - > dedicated to retrieving screts from the Secret Backend. I discussed it > with Ash > and Kamil > <https://apache-airflow.slack.com/archives/C0145R4NPS5/p1589805908013700> on > Slack today. So far I thought I treat them as usual providers, but Ash > raised some valid concenrs. so I wanted to raise teh proposal before I > start working on it/ > > *Context:* > > Currently we have "Secret Backend" support built in in 2.0 and > 1.10.10+. It > includes retrieving the variable and connections (via Secret Manager class) > for: > > - Hashicorp Vault > - Secret Manager > - KMS > - AWS secret manager > > Those secret managers are configured in: > > [secret] > backend=<SecretManagerClass> > backend_kwargs={} > > Those are available for use in a nice way (via Jinja templates and the > like), but they need support in the Core of Airlfow (so require 1.10.10+). > This means that if you are on pre 1.10.10 you cannot use those secrets. > Currently you can only use one secret per whole Airflow installation > so if > your secrets are split between several secret managers (or if secrets for > particular service require different credentials) - you cannot use the > mechanism to access such distributed secrets. It's not often case, but I > very well imagine it might happen that there are different sets of > credentials to access different secrets - some services might have > different scopes/level of access needed. . > > *Proposal* > > We have an idea that we might want also (on top of the above SecretManager > implementation) define generic Hooks for accessing secrets from those > services (just generic secrets, not connection, variables). Simply treat > each of the backends above as another "provider" and create a Hook to > access the service. Such Hook could have just one method: > > def get_secret(self, path_prefix: str, secret_id: str) -> Optional[str] > > It would use a connection defined (as usual) in ENV variables or database > of Airflow to authenticate with the secret service and retrieve the > secrets. > > The good thing with it is that you could have easily multiple secret > backends configured to retrieve secrets for specific "service" (so > that you > could keep "generic airflow's secerts" in one backend but still have > possibility of custom operators to use other backends (with different > authentication, scopes etc.). And it is not touching any of the > "core" of > Airflow. It's just a set of hooks with corresponding connections that work > the same way as accessing any other provider in Airflow. No core of Airflow > will be touched with this change. > > *Pros/Cons* > > *Con:* > > I do realise it is a bit of duplication in functionality. We already > have a > way to connect to a secret backend via airflow configuration and we should > likely promote it rather than introduce additional mechanism. > > *Pros:* > > * Most of all -> it adds flexibility of accessing several secret backends > for different use-cases. I looked at it so far in the way those hooks are > merely another set of "provider hooks". For me this is nothing different > than "providers" for any other services we have. fFr example "cloudant" > provider has only "CloudantHook" that other custom operators can use. > And I > well imagine this might be actually even more convenient to configure > connections in the DB and access secrets this way rather than having to > configure Secret Backends in Airflow configuration. > > * The dupication there it is very, very limited (basically a method > call to > secret backend). > > * Another benefit of it is that it would allow people still stuck on pre > 1.10.10 to write custom operators that would like to use secret backends > (via backport operators). And still continue doing it in the future > (possibly migrating to 2.0/1.10.10+ in cases when there is one secret > backed only - but continue ot use connections/hooks where some specific > secrets shoudl be kept in different secret backend. > > I would like to hear your opinion on that. > > J. > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> >
