I find working with sensitive Connections in Airflow quite difficult. In simple words - once a connection is defined anyone who knows about it can use it. This is a problem when you work with sensitive data like HR or finance. The issue is not about storing the connection details securely but rather once defined - who can use it? how to prevent from unauthorized users to access it? I would love to see the concept of namespace/areas in Airflow so Pools/Connections/Variables/Dags and user login are all associated to a specific namespace/areas. Kinda similar to the namespace concept in K8s I guess.
For the moment we solved it by having two separate Airflow instances (one regular and one for sensitive data) but this is very difficult to maintain. On Tue, Jun 15, 2021 at 12:54 PM Ash Berlin-Taylor <[email protected]> wrote: > Hi everyone, > > As I'm sure many of you are aware I (along with Aizhamal) am giving the > opening keynote at this year's Airflow summit, and I'm covering "what's > next after 2.0" -- essentially what is the roadmap for Airflow for the next > 12-18 months. > > Since Airflow is a community project first and foremost I'd like to get > all your ideas, no matter how off the wall :) > > I've got my own ideaas, and 2.2 is fairly firm already (AIPs 39 and 40), > but 2.3 and beyond starts to get less clear, so if you have something that > you'd like to see Airflow be able to do or do better, now is the time to > speak up. > > You don't have to have a solution, just "I find doing X > hard/annoying/difficult" is enough. > > (And a general reminder: the roadmap is a statement of intent, not a > promise of timeline or even that a feature will actually be implemented) > > To keep this thread manageable, please can we avoid discussions _*in this > thread*_ about ideas and keep +1/me too's to a minimum. > > Cheers, > -ash >
