FrankYang0529 opened a new pull request, #72858:
URL: https://github.com/apache/airflow/pull/72858
## Why
- An executor or auth manager configured by module path, without being
packaged as a provider, loses every CLI command it defines in
`get_cli_commands()`. The commands are missing with no warning and no error.
- Before the provider `cli` section was introduced (#59805), `cli_parser`
looped over every configured executor and called the configured auth manager.
- `core-concepts/executor/index.rst` and
`core-concepts/auth-manager/index.rst` still document `get_cli_commands()`, and
the auth manager page still shows `[core] auth_manager =
my_company.auth_managers.MyCustomAuthManager` as the way to configure a custom
class.
## How
- Build the set of class paths a provider `cli` section already covers, skip
those, and send everything else through the fallback.
- Skip classes shipped in airflow-core by module prefix, so the default
configuration never imports them.
- Give commands registered by a provider priority over the fallback. A
subclass of a provider executor or auth manager inherits `get_cli_commands()`.
Without this, it registers the same names a second time. `cli_parser` raises
`CliConflictError` on a duplicate name and the CLI fails to start.
## Verification
- Unit test: `uv run --project airflow-core pytest
airflow-core/tests/unit/cli/test_cli_parser.py -q`
- Integration test:
1. Setup
```sh
mkdir -p files/demo/my_company
touch files/demo/my_company/__init__.py
cat > files/demo/my_company/executors.py <<'EOF'
from airflow.cli.cli_config import ActionCommand, GroupCommand
from airflow.executors.base_executor import BaseExecutor
from airflow.providers.celery.executors.celery_executor import CeleryExecutor
def ping(args):
print("pong from MyExecutor")
class MyExecutor(BaseExecutor):
"""Plain custom executor: implements BaseExecutor, not packaged as a
provider."""
@staticmethod
def get_cli_commands():
return [
GroupCommand(
name="my-executor",
help="Manage MyExecutor",
subcommands=[ActionCommand(name="ping", help="Ping",
func=ping, args=())],
)
]
class MyCeleryExecutor(CeleryExecutor):
"""Custom executor built on a provider executor: inherits
CeleryExecutor.get_cli_commands()."""
EOF
cat > files/demo/my_company/auth_managers.py <<'EOF'
from airflow.api_fastapi.auth.managers.simple.simple_auth_manager import
SimpleAuthManager
from airflow.cli.cli_config import ActionCommand, GroupCommand
from airflow.providers.fab.auth_manager.fab_auth_manager import
FabAuthManager
def whoami(args):
print("hello from MyCustomAuthManager")
class MyCustomAuthManager(SimpleAuthManager):
"""Plain custom auth manager, not packaged as a provider."""
@staticmethod
def get_cli_commands():
return [
GroupCommand(
name="my-auth",
help="Manage MyCustomAuthManager",
subcommands=[ActionCommand(name="whoami", help="Who am I",
func=whoami, args=())],
)
]
class MyFabAuthManager(FabAuthManager):
"""Custom auth manager built on a provider auth manager: inherits
FabAuthManager.get_cli_commands()."""
EOF
```
```sh
uv sync --frozen --project airflow-core
export AIRFLOW_HOME="$(pwd)/files/demo/home"
export AIRFLOW__CORE__LOAD_EXAMPLES=False
export PYTHONPATH="$(pwd)/files/demo"
```
2. Check custom executor
```sh
AIRFLOW__CORE__EXECUTOR=my_company.executors.MyExecutor .venv/bin/airflow
--help 2>/dev/null | grep -c my-executor
AIRFLOW__CORE__EXECUTOR=my_company.executors.MyExecutor .venv/bin/airflow
my-executor ping
```
On main branch, the count is `0`. On this branch, it shows `1` and `pong
from MyExecutor`.
3. Check custom auth manager
```sh
AIRFLOW__CORE__AUTH_MANAGER=my_company.auth_managers.MyCustomAuthManager
.venv/bin/airflow --help 2>/dev/null | grep -c my-auth
AIRFLOW__CORE__AUTH_MANAGER=my_company.auth_managers.MyCustomAuthManager
.venv/bin/airflow my-auth whoami
```
On main branch, the count is `0`. On this branch, it shows `1` and `hello
from MyCustomAuthManager`.
4. Check classes inheriting from a provider
```
AIRFLOW__CORE__EXECUTOR=my_company.executors.MyCeleryExecutor
.venv/bin/airflow --help 2>/dev/null | grep -cE '^\s+celery\s'
AIRFLOW__CORE__AUTH_MANAGER=my_company.auth_managers.MyFabAuthManager
.venv/bin/airflow --help 2>/dev/null | grep -cE '^\s+users\s'
```
These two are regression guards rather than bug demos. On main they also
print 1, because the fallback that imports the class and calls
`get_cli_commands()` never runs there. What they prove is that re-opening that
fallback does not start registering duplicate commands.
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull
request.
Write a good git commit message following this guide:
https://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the
documentation.
For user-facing UI changes, please attach before/after screenshots (or a
short
screen recording) so reviewers can assess the visual impact.
Feel free to ping (in general) for the review if you do not see reaction for
a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we
sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change below checkbox to `[X]` followed by the name of the tool, uncomment
the "Generated-by".
-->
- [X] Yes - Claude Code
<!--
Generated-by: [Tool Name] following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
