potiuk commented on PR #60074: URL: https://github.com/apache/airflow/pull/60074#issuecomment-3710183909
And just to explain a bit more context: > Then it is really a matter of proper sequence of initialization in each of the components And once we do that now (which should not be that difficult to be honest) - we **might** be able to get rid of those pesky `provider_config_fallback_defaults.cfg` - the main reason for having those defaults, was that the provider's configuration initilaization could be done **after** those defaults were needed from providers. So for example in scheduler, kubernetes configuraitoin could be read **before** ProvidersManager() was initialized, and we needed defauly values for that params in case they were not defined locally. However, if we do the initialization first, and in a controlled way and inject the defaults retrieved from providers, this will not be needed any more. I think one of the critical places there are CLi parameters, on one hand, some of them are reading the values from config but they really should be read before ProvidersManager() was initialized. This might however also change with https://github.com/apache/airflow/pull/59805 of @jason810496 -> because with that change (after performance testing) it turns out that we can initialize ProvidersManager() partially when we are parsing CLI - without initializing all of it it as provider's manager has partial lazy initialization built-in. Once this is done, I think the main reason where we could not have provider's manager initialization done realatively early and before all places we needed - will be gone, and we might get rid of those pesky fallback_defaults. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
