samraj2k commented on issue #61645: URL: https://github.com/apache/airflow/issues/61645#issuecomment-5003604785
> I think that deserves a devlist discussion to start with. > > While I understand the need to manage such bundle configuration, I do not think personally this is necessary good idea to do it via REST API. If you look at our (updated wiht 3.2.0) security model https://airflow.apache.org/docs/apache-airflow/stable/security/security_model.html - this approach introduce a compleletly new ,security realm for Airflow. It's been vastly updated now and refreshed for Airflow 3.2 and you are welcome to take a deep look there and understand some basica assumptions of security principles and isolation between roles, components and workflows that we have in Airflow. This should be a good and deep reading before starting discussion on devlist - because this will be the first question you will get asked when you raise such proposal in devlist. > > So Phase 2 in your case is not even nearly enough - and it should be rather Phase 0. We need to know what kind of security realm we are proposing, what kind of roles it creates for Airflow and whether we want to take some risks in Airlfow for moving part of the currently delegated security responsibility (to deployment and Deployment Manager) - to be handled by Airflow software. > > Currrently bundle definition, access to Dags and configuration of Airflow - including where Dags are retrieved from, what is the bundle definition etc. is strictly limited to "Deployment Manager" role. Deployment Manager uses Deployment-specific mechanisms to set the configuration and manage it - and it's not exposed or managed for security from Airflow "software" point of view - it is left entirely to Deployment Manager and all kinds of security mechanisms that are specific to deployment type chosen by Deployment Manager. > > By adding Dag Bundle configuration, we are extending the model with a new Role - someone who can manage part of Airflow configuration - boundle definition - via API and api -server. > > This has far-reaching implication. For example currently api-server can be run in an environment where airflow configuration - including bundle definition is read only. And this is a built-in security feature - it makes it simply impossible, even if api-server security is broken, to permanently change configuration of Airflow remotely via REST API . > > So while I am not 100% against such feature - it has to be first discussed in the devlist and it should be - eventually voted on - likely with a complete AIP describing security implications, new roles proposed, scope and security boundaries of such change. Thanks @potiuk, we have a similar use case, and I agree that writable bundle configuration through the Airflow API would introduce a new security realm. We’ are proposing following change - Introduce a BaseDagBundleConfigProvider abstraction used by DagBundlesManager.get_all_dag_bundles(). Reference: https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/dag_processing/bundles/manager.py#L417 - Keep the current static configuration as the default provider/behavior. - Allow Deployment Managers to configure a custom provider whose responsibility will be to get all the bundles on every DagProcessor loop. - Update the DAG processor to periodically reload all bundles (in main DagProcessor loop) and reconcile newly discovered bundles without restart. This way, we wont we creating any new write APIs. The logic to fetch the bundles (on which we do reload) resides outside of Airflow, so users can fetch the bundles either from external source like S3, any API etc This also solves for: https://github.com/apache/airflow/discussions/59799 by being generic. Let me know how this sounds and the next steps. Would a dev-list [DISCUSS] thread/AIP be the appropriate next step. I believe the hook plus DAG processor reload address the security concern you have mentioned. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
