potiuk commented on issue #9342: URL: https://github.com/apache/airflow/issues/9342#issuecomment-899674097
> This is like a key feature. I can't believe we currently don't have a way to set permissions to a group of DAGs. Tags may not be the best, but I think it could work, like "You can only see DAGs tagged with data-science tag". My personal opninon - and this is not a universal truth, but my persona view on this is, that this is not really whether we want to use tags or something else. I think main reason is because in essence multi-tenancy feature of Airflow does not have a chance to be working (yet!). By providing "access per-group" UI feature, we would give people a completely false impression, that you can isolate those DAG groups from each other. I personally believe it would be a very bad idea to tell people they can do it, while under the hood they would miss the isolation. In short - you CAN't currently isolate groups of DAGs in single instance of Airflow. You simply CAN't. it's impossible by design, and we are aiming to iterate and improve the design in the future to make it possible, but it will take a while (and likely will only be fully available in Airflow 3). There are multiple reasons why this kind of RBAC control is a UI-only feature (and can't be easily made into "execution isolation" - while "Role" separation makes sense as UI-only feature (Admin can access different functionality than User via UI). I personally think introducing groups of DAGs is only really useful (and secure) if we will be able to isolate DAG groups not only via "UI" capabilities but also via "DAG writing" capabilities. I think the reason why you want to introduce this in UI is tha there is something that you want to prevent people doing between the groups (viewing, running, etc.). I think the main reason to give UI access for group is to give it to the same people who can write the dags. For me pretty much all UI "Can see DAGs in this group" access should map 1-1 to "Can write DAGS in this group". It's a bit simplistic view, I can - of course - imagine other reasons for permission separation, but to be honest - giving people the way to "view and operate" DAGs via UI "per-group", and not doing the same for "write code in DAGs" gives a very bad false-sense of security. And I think we should only introduce it when we can do it "full-stack". Otherwise people will be tempted to use current airflow as multi-ten ant, where it is in fact not multi-tenant (yet!). Currently the only way to have multi-tenancy is to have separate airflow instances for each tenant. Full stop. By introducing DAG grouping without going full-stack we give people wrong impression that the story is different. It's not. > Since this has not been added into 2.1, when are we expecting 3.0 to be launched? Can't we expect this to be merged sooner? I think it will take a while. If you feel the need that you MUST separate the DAGs in the UI, I'd strongly encourage you to implement single instance od Airflow per "group". This is far more robust, already proven, It provides better isolation between groups - actually it's infinitely bigger - comparing to trying to run them in single instance which provides no isolation. Better yet, it provides also capability of isolating the workloads between the different groups (which is completely not possible if you run them in single instance and one group can submit million tasks influencing the other). It can be nicely integrated into any authentication mechanism you have in the way that only users belonging to certain group can have access to each instance. And it can be easily automated to create/drop new "instances" automatically for different groups (been there, done that with nice Terraform templates) - even if you need to manage many of those. Yeah it uses more resources, but you can also reuse a lot of those resources if you put those instances in auto-scaling K8S clusters. And you can create separate schema for each "group" in the same database to utilise your DB server better. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
