potiuk opened a new pull request, #32604:
URL: https://github.com/apache/airflow/pull/32604
The changes implemented:
* provider.yaml files for providers can optionally contribute extra
configuration, the configuration is exposed via "get_provider_info" entrypoint,
thus allowing Airflow to discover the configuration from both - sources (in
Breeze and local development) and from installed packages
* Documentation for configuration contributed by providers is generated as
part of Provider documentation. It is also discoverable by having a
"core-extension" page displaying all community providers that contribute their
own configuration.
* Celery configuration (and in the future Kubernetes configuration) is
linked directly from the airflow documentation - the providers are
preinstalled, which means that celery (and Kubernetes in the future)
configuration is considered as important to be directly mentioned and linked
from the core. Similarly Celery and Kubernetes executor documentation remains
in the core documentation (still configuration options are detailed only in the
provider documentation and only linked from the core.
* the `airflow config list` command becomes more powerful, with new flags
showing optionally decriptions, examples, environment variables corresponding
to the commands, showing sources where the configuration value comes from
(including provider information and also allowing to show just default values -
which nicely replaces the "default_airflow.cfg"
In order to that, howver, some crucial changes had to be done to refactor
the code for configuration retrieval.
This change refactors configuration retrieval quite heavily in order to
allow the default configuration to be contributed by providers.
It simplifies and removes some of the historical internals of configuration
reading and writing, removing unnecessary and unused code and finally replacing
the mechanism where we kept configuration description in both yaml files and
pre-rendered template files, thus removing the need of keeping the rendered
configuration in the airflow package.
* config.yml becomes the only source of truth of default configuration
values and configuration description (including comments) for Airflow core
* default_airflow.cfg and default_test.cfg templates are deleted and
default_airflow.cfg content is replaced with comments explaining how to get
default config (with airflow config defaults command). Those templates were
confusing because they almost looked like airflow configuration but they
weren't (they contained JINJA directives)
* unittest.cfg is not used any more - test configuration is dynamically
generated when tests are starting with pytest fixture. The source of truth for
test configuration is config_unit_test.yml
* removed airflow_ci.cfg which was yet another configuration file used only
in CI - yet another source of confusion. All the test configuration needed is
now part of config_unit_tests.yml. Thanks to that, unit tests for configuration
work now in CI and local venv without having to set special environment
variables or configuration
* old methods of processing the default_*.cfg and writing the config have
been removed as we do not have the JINJA templates any more for those.
* the code that generates documentation from configuration has been
refactored to avoid unnecessary repetitions
* secret configuration tests using Hashicorp have been moved to Hashicorp
provider, thanks to that we can run config tests without installing hvac.
* there were test covering type coertion for configuration - they were using
"kerberos" configuration but thanks to config_unit_tests.yml approach,
converted them to "abstract" tests
* moved default_celery.py configuration to provider (this is a remnant of
the celery executor moving to provider. There is no need for backwards
compatibility and PEP 562, it is only used by Celery* executors.
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
