potiuk commented on code in PR #32628: URL: https://github.com/apache/airflow/pull/32628#discussion_r1264609814
########## airflow/providers/celery/executors/default_celery.py: ########## Review Comment: > Personally, I am not a fan of relying on Airflow configurations from providers. The reason for my stance is that providers can be installed on various versions of Airflow, and if we add a new feature relies on a new configuration, it may not be supported by older versions of Airflow. IMHO, it would be beneficial to explore alternative methods for configuring executors with a minimal usage of Airflow configurations. I believe when you look in detail to where #32604 is heading, you will see the goal achieved and you will see that what we got there, we achieve everything you wanted, without breaking compatibility with the current system and with 100% reuse of what we already have as various mechanisms where configuration can be set. I think this is actually the goal of https://github.com/apache/airflow/pull/32604 when (after I extract various parts of it it to separate PRs and merge them, we are heading up. It's not by introducing a different mechanism for providers, but by making sure that the configuration they use is tied with the provider version not airflow version Generally speaking, in providers we will only use the "framework" from airfow. basically `confg.get()` which is supported by any and all versions of Airflow and the whole mechanism of being able to set it via various mechanisms: file, env, secret backend etc.. Everything else will be separately controlled in each provider. What #32604 does (and very gently without breaking any compatiblities). * Each provider might be able to use the set of configurations (group/param) that they are bound to and use - and will be able to separately add or remove configuraiton in whatever provider (not Airflow). So provider, for each version will use the same set of parameters - regardless from the version of Airflow it is installed in. * Each provider version will have embedded (as provider.yaml/get_provider_info entrypoint) description of the configuration they support. Each version of provider will have their own variant of it. If you add configuration in version 3.1.0, version 3.0.0 will not have/use it * Each provider which has its own configuration will have their own dedicated configuration page (see screnshots in #32604). And it also will be versioned. You will be able to switch the version and you will be able to see which set of configuration parameters is supported by each version. * You will even be able to see version_added there (which will be version of the provider, not airflow) * Those provider configuration variables will NOT be shown at the main https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html - they will be linked from there via https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/index.html (there will be dedicated page which will show which providers have configurations and link to their respective (stable) versions - but from there you will be able to see historically how configuration parameter changed for that provider. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
