amoghrajesh commented on issue #44352: URL: https://github.com/apache/airflow/issues/44352#issuecomment-3219718374
Alright, nice discussions above. I would be in favour of _**Option 1: Complete Independence**_ as I think it is the most practical and architecturally sound approach with Airflow3 architecture. Let me explain the fundamental problems that eliminate Options 2, 3, and 4 from consideration: * Option 2: API-Based Inheritance It creates network dependencies between the workers and the API server and now makes the workers running task sdk dependent on API server even for the most basic dag authoring experience, I do not know how we would be able to deal with operator and dag defaults here, not to mention it might turn into really huge payloads across the networks making a mark on performance of Airflow by having too many API calls and also by making "context" as a dumping groung. * Option 3: Bootstrap Injection Could potentially lead into slower bootstrapping for configs that need to be calculated and it still requires API calls and doesn't solve the DAG processor/triggerer problem. Another issue I see is static nature of injected configuration means no dynamic updates are possible, leading to inconsistency when configuration changes on the server side while tasks are running (could be an issue for things like execution url changing) * Option 4: Hybrid Local-First + API Fallback This is a decent option and my proposal would be more like a hybrid of option 4 and 1, but 4 in itself could be overly complex for very little benefit? I propose that we move the `AirflowConfigParser` (not configs) to a shared module, while keeping the actual configuration files and values separate and independent for each component. Each component now gets exactly what it needs - the shared parsing logic / abstraction to read and interpret configuration files, but with complete autonomy over what configuration values they actually contain. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
