potiuk edited a comment on issue #19251: URL: https://github.com/apache/airflow/issues/19251#issuecomment-958956103
I would not close @ashb, but rather discuss how we can implement, I think there are varying opinions here. At the very least, if others also have such strong opinion as you - we should document how to handle this - very common it seems - use case. I think we still can discuss how to do it - and maybe we should do it differently - but I think the use/case and rationale is very-sound and - more than that - pretty common. It has been raised quite a few times by different people. Since we have connections where most people keep their secret passwords, I think the ratio of secret variables vs. the non-secret ones is small and this use case is very common. I'd say more often than not people will have connections kept in secret, but they will have no secret variables. We are basically telling people to pay more for their secret backends and we do not even tell them how to avoid this. And I think making people write their onwn secret backend to handle such case is kinda over-the-top - especially that it seems common pattern for all the different secret backends. And our users do not often realise, that they could do their own implementation very easily. I actually see the point of the "different access" pattern that @ashb and I agree it is not perfect. I think what is even bigger problem is that it's the "DAG" writers to decide how each variable should be retrieved, which - I quite agree - is very prone to different kinds of problems But maybe we can find a a good solution that serves the use case but does not introduce the "different access" pattern. I Maybe a good solution will be to give them ready-to use simple implementation of such custom backend that you could "MixIn" with any other secret backend where for example chosing whether to use secret or not would be done based on Regexp match of the variable name? Maybe simply have a possibility to define an optional callable configurable in `secrets` group, that will get the "name" and "type" (variable/connection/config) of the retrieved object and return true/false if it should use secret ? That could also give an opportunity to handle a different case we have currently problem with. Currentlly when someone tries to set a variable whcih is already available as "secret". this variable is set in the Metadatada DB (and not accessible). You do not get any warning or error when you do so, which might lead to really strange problems. If the installation has this "callable" defined however, that could be much better, because we could check whether the variable should be read from secret and if so - we would just fail such an attempt. I think that would be a really nice and consistent approach. WDYT everyone? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
