potiuk commented on issue #19251:
URL: https://github.com/apache/airflow/issues/19251#issuecomment-958956103
I would not close @ashb, but rather discuss how we can implement, I think
there are varying opinions here. At thev very least) if others also have such
strong opinion as you - we shoudl document how to handle this - very common it
seems use case.
I think we still can discuss how to do it - and maybe we should do it
differently - but I think the use/case and rationale is very-sound and - more
than that - pretty common.
I think this issue has been raised quite a few times by different people.
Since we have connections where, I think the ratio of secret variables vs.
the non-secret ones is small and this use case is very common. I'd say more
often than not people will have connections kept in secret, but they will have
no secret variables. We are basically telling people to pay more for their
secret backends and we do not even tell them how to avoid this. And I think
making people write their onwn secret backend to handle such case is kinda
over-the-top - especially that it seems common pattern for all the different
secrets. And our users do not often realise, that they could do their own
implementation very easily.
I actually see the point of the "different access" pattern that @ashb and I
agree it is not perfect. I think what is even bigger problem is that it's the
"DAG" writers to decide how each variable should be retrieved, which - I quite
agree - is very prone to different kinds of problems
But maybe we can find a a good solution that serves the use case but does
not introduce the "different access" pattern. I
Maybe a good solution will be to give them ready-to use simple
implementation of such custom backend that you could "MixIn" with any other
secret backend where for example chosing whether to use secret or not would be
done based on Regexp match of the variable name?
Maybe simply have a possibility to define an optional callable configurable
in `secrets` group, that will get the "name" and "type"
(variable/connection/config) of the retrieved object and return true/false if
it should use secret ? That could also give an opportunity to handle a
different case we have currently problem with. Currentlly when someone tries to
set a variable whcih is already available as "secret". this variable is set in
the Metadatada DB (and not accessible), you do not get any warning or error
when you do so, which might lead to really strange problems.
If the installation has this "callable" defined however, that could be much
better, because we could check whether the variable should be read from secret
and if so - we would just fail such an attempt.
I think that would be a really nice and consistent approach.
WDYT everyone?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
