1fanwang opened a new pull request, #73743:
URL: https://github.com/apache/airflow/pull/73743
`airflowctl dags get-tags` crashes as soon as any Dag has a tag, whichever
`--output` is chosen:
```
ValueError: To tabulate non-dictionary data you need to provide `mapper`
function
```
## Why
The `dagTags` endpoint returns tag names as plain strings, for example
`{"tags": ["etl", "nightly"], "total_entries": 2}`. Generated commands drop
`total_entries` and print the one list that remains, but
`AirflowConsole.print_as` only renders records and raises for anything else.
Among the API's collection responses, `DAGTagCollectionResponse` is the only
one whose list holds plain values, so `get-tags` is the only command that hits
this. It works only while no Dag has a tag, because an empty list passes the
check. The 0.1.5 release fails the same way.
## What
When the result to print is a list of plain values, airflowctl now prints
one row per value, keyed by the field name. `dags get-tags -o json` prints
`[{"tags": "etl"}, {"tags": "nightly"}]`, and the table and plain formats print
one tag per line under a `tags` header. Lists of records, and values nested
inside a record such as a Dag run's `conf`, keep their current output.
The new unit test runs `dags get-tags` through the real argument parser for
all four output formats and fails on `main` with the error above. The command
is also added to the airflow-ctl integration tests, whose example Dags have
tags.
## Testing Done
airflowctl built from `main` at 28cab90ccb and from this branch, each run
against a real Airflow 3.3.2 API server with a fresh SQLite database and two
tagged Dags. The script in the details block starts the server, signs in with
`airflowctl auth token`, and asks for the tags in each output format.
| Output | `main` | This branch |
|---|---|---|
| `json` | `ValueError`, exit 1 | `[{"tags": "etl"}, {"tags": "nightly"},
{"tags": "reporting"}]`, exit 0 |
| `yaml` | `ValueError`, exit 1 | one `- tags:` item per tag, exit 0 |
| `table` | `ValueError`, exit 1 | one tag per line under `tags`, exit 0 |
| `plain` | `ValueError`, exit 1 | one tag per line under `tags`, exit 0 |
Setup, on macOS with Python 3.12:
```bash
# on main at 28cab90ccb, then again on this branch with -o
/tmp/gt-wheels/fix and /tmp/gt-ctl-fix
uv build --package apache-airflow-ctl --wheel -o /tmp/gt-wheels/main
uv venv --python 3.12 /tmp/gt-ctl-main
uv pip install --python /tmp/gt-ctl-main/bin/python
/tmp/gt-wheels/main/apache_airflow_ctl-0.1.5-py3-none-any.whl
# the server
uv venv --python 3.12 /tmp/gt-af332
uv pip install --python /tmp/gt-af332/bin/python "apache-airflow==3.3.2" \
--constraint
https://raw.githubusercontent.com/apache/airflow/constraints-3.3.2/constraints-3.12.txt
```
Before, with airflowctl from `main`:
```
$ bash ./get-tags-e2e.sh /tmp/gt-ctl-main/bin/airflowctl
/tmp/gt-af332/bin/airflow
$ airflow version # the server
3.3.2
$ airflowctl auth login --api-url http://127.0.0.1:58137 --skip-keyring #
AIRFLOW_CLI_TOKEN is set
Login successful! Welcome to airflowctl!
$ curl -s -H "Authorization: Bearer $AIRFLOW_CLI_TOKEN"
http://127.0.0.1:58137/api/v2/dagTags
{"tags":["etl","nightly","reporting"],"total_entries":3}
$ airflowctl dags get-tags -o json
Traceback (most recent call last):
File "/tmp/gt-ctl-main/bin/airflowctl", line 10, in <module>
sys.exit(main())
^^^^^^
File
"/tmp/gt-ctl-main/lib/python3.12/site-packages/airflowctl/__main__.py", line
34, in main
safe_call_command(args.func, args=args)
File
"/tmp/gt-ctl-main/lib/python3.12/site-packages/airflowctl/ctl/cli_config.py",
line 78, in safe_call_command
function(args)
File
"/tmp/gt-ctl-main/lib/python3.12/site-packages/airflowctl/api/client.py", line
563, in wrapper
return func(*args, api_client=api_client, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/tmp/gt-ctl-main/lib/python3.12/site-packages/airflowctl/ctl/cli_config.py",
line 933, in _get_func
AirflowConsole().print_as(
File
"/tmp/gt-ctl-main/lib/python3.12/site-packages/airflowctl/ctl/console_formatting.py",
line 125, in print_as
raise ValueError("To tabulate non-dictionary data you need to provide
`mapper` function")
ValueError: To tabulate non-dictionary data you need to provide `mapper`
function
exit=1
$ airflowctl dags get-tags -o yaml
Traceback (most recent call last):
File "/tmp/gt-ctl-main/bin/airflowctl", line 10, in <module>
sys.exit(main())
^^^^^^
File
"/tmp/gt-ctl-main/lib/python3.12/site-packages/airflowctl/__main__.py", line
34, in main
safe_call_command(args.func, args=args)
File
"/tmp/gt-ctl-main/lib/python3.12/site-packages/airflowctl/ctl/cli_config.py",
line 78, in safe_call_command
function(args)
File
"/tmp/gt-ctl-main/lib/python3.12/site-packages/airflowctl/api/client.py", line
563, in wrapper
return func(*args, api_client=api_client, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/tmp/gt-ctl-main/lib/python3.12/site-packages/airflowctl/ctl/cli_config.py",
line 933, in _get_func
AirflowConsole().print_as(
File
"/tmp/gt-ctl-main/lib/python3.12/site-packages/airflowctl/ctl/console_formatting.py",
line 125, in print_as
raise ValueError("To tabulate non-dictionary data you need to provide
`mapper` function")
ValueError: To tabulate non-dictionary data you need to provide `mapper`
function
exit=1
$ airflowctl dags get-tags -o table
Traceback (most recent call last):
File "/tmp/gt-ctl-main/bin/airflowctl", line 10, in <module>
sys.exit(main())
^^^^^^
File
"/tmp/gt-ctl-main/lib/python3.12/site-packages/airflowctl/__main__.py", line
34, in main
safe_call_command(args.func, args=args)
File
"/tmp/gt-ctl-main/lib/python3.12/site-packages/airflowctl/ctl/cli_config.py",
line 78, in safe_call_command
function(args)
File
"/tmp/gt-ctl-main/lib/python3.12/site-packages/airflowctl/api/client.py", line
563, in wrapper
return func(*args, api_client=api_client, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/tmp/gt-ctl-main/lib/python3.12/site-packages/airflowctl/ctl/cli_config.py",
line 933, in _get_func
AirflowConsole().print_as(
File
"/tmp/gt-ctl-main/lib/python3.12/site-packages/airflowctl/ctl/console_formatting.py",
line 125, in print_as
raise ValueError("To tabulate non-dictionary data you need to provide
`mapper` function")
ValueError: To tabulate non-dictionary data you need to provide `mapper`
function
exit=1
$ airflowctl dags get-tags -o plain
Traceback (most recent call last):
File "/tmp/gt-ctl-main/bin/airflowctl", line 10, in <module>
sys.exit(main())
^^^^^^
File
"/tmp/gt-ctl-main/lib/python3.12/site-packages/airflowctl/__main__.py", line
34, in main
safe_call_command(args.func, args=args)
File
"/tmp/gt-ctl-main/lib/python3.12/site-packages/airflowctl/ctl/cli_config.py",
line 78, in safe_call_command
function(args)
File
"/tmp/gt-ctl-main/lib/python3.12/site-packages/airflowctl/api/client.py", line
563, in wrapper
return func(*args, api_client=api_client, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/tmp/gt-ctl-main/lib/python3.12/site-packages/airflowctl/ctl/cli_config.py",
line 933, in _get_func
AirflowConsole().print_as(
File
"/tmp/gt-ctl-main/lib/python3.12/site-packages/airflowctl/ctl/console_formatting.py",
line 125, in print_as
raise ValueError("To tabulate non-dictionary data you need to provide
`mapper` function")
ValueError: To tabulate non-dictionary data you need to provide `mapper`
function
exit=1
```
After, with airflowctl from this branch:
```
$ bash ./get-tags-e2e.sh /tmp/gt-ctl-fix/bin/airflowctl
/tmp/gt-af332/bin/airflow
$ airflow version # the server
3.3.2
$ airflowctl auth login --api-url http://127.0.0.1:58193 --skip-keyring #
AIRFLOW_CLI_TOKEN is set
Login successful! Welcome to airflowctl!
$ curl -s -H "Authorization: Bearer $AIRFLOW_CLI_TOKEN"
http://127.0.0.1:58193/api/v2/dagTags
{"tags":["etl","nightly","reporting"],"total_entries":3}
$ airflowctl dags get-tags -o json
[{"tags": "etl"}, {"tags": "nightly"}, {"tags": "reporting"}]
exit=0
$ airflowctl dags get-tags -o yaml
- tags: etl
- tags: nightly
- tags: reporting
exit=0
$ airflowctl dags get-tags -o table
tags
=========
etl
nightly
reporting
exit=0
$ airflowctl dags get-tags -o plain
tags
etl
nightly
reporting
exit=0
```
The released 0.1.5 from PyPI, installed with `uv pip install --python
/tmp/gt-ctl-pypi/bin/python apache-airflow-ctl==0.1.5`, fails the same way:
```
$ bash ./get-tags-e2e.sh /tmp/gt-ctl-pypi/bin/airflowctl
/tmp/gt-af332/bin/airflow > release-0.1.5.log 2>&1
$ grep -E '^\$ airflowctl dags|^exit=|^ValueError' release-0.1.5.log
$ airflowctl dags get-tags -o json
ValueError: To tabulate non-dictionary data you need to provide `mapper`
function
exit=1
$ airflowctl dags get-tags -o yaml
ValueError: To tabulate non-dictionary data you need to provide `mapper`
function
exit=1
$ airflowctl dags get-tags -o table
ValueError: To tabulate non-dictionary data you need to provide `mapper`
function
exit=1
$ airflowctl dags get-tags -o plain
ValueError: To tabulate non-dictionary data you need to provide `mapper`
function
exit=1
```
<details>
<summary>Reproducer source: get-tags-e2e.sh</summary>
```bash
#!/usr/bin/env bash
# List Dag tags with airflowctl against a real Airflow API server that has
two tagged Dags.
# Usage: get-tags-e2e.sh <airflowctl executable> [airflow executable for the
server]
set -u
CTL=${1:?usage: get-tags-e2e.sh <airflowctl> [airflow]}
AIRFLOW=${2:-airflow}
PORT=$(python3 -c 'import socket; s = socket.socket(); s.bind(("127.0.0.1",
0)); print(s.getsockname()[1])')
URL="http://127.0.0.1:$PORT";
SERVER_HOME="$(mktemp -d)"
CLIENT_HOME="$(mktemp -d)"
SERVER_ENV=(
AIRFLOW_HOME="$SERVER_HOME"
AIRFLOW__CORE__LOAD_EXAMPLES=False
AIRFLOW__CORE__DAGS_FOLDER="$SERVER_HOME/dags"
AIRFLOW__CORE__SIMPLE_AUTH_MANAGER_USERS=admin:admin
)
say() { printf '\n$ %s\n' "$1"; }
mkdir "$SERVER_HOME/dags"
cat > "$SERVER_HOME/dags/tagged.py" <<'EOF'
from airflow.sdk import dag, task
@dag(schedule=None, tags=["etl", "nightly"])
def orders():
@task
def load():
pass
load()
@dag(schedule=None, tags=["etl", "reporting"])
def revenue():
@task
def report():
pass
report()
orders()
revenue()
EOF
env "${SERVER_ENV[@]}" "$AIRFLOW" db migrate > "$SERVER_HOME/setup.log" 2>&1
|| { cat "$SERVER_HOME/setup.log"; exit 1; }
env "${SERVER_ENV[@]}" "$AIRFLOW" dags reserialize >>
"$SERVER_HOME/setup.log" 2>&1 || {
cat "$SERVER_HOME/setup.log"
exit 1
}
env "${SERVER_ENV[@]}" "$AIRFLOW" api-server --port "$PORT" --workers 1 >
"$SERVER_HOME/api-server.log" 2>&1 &
SERVER_PID=$!
trap 'kill "$SERVER_PID" 2>/dev/null; wait "$SERVER_PID" 2>/dev/null; rm -rf
"$SERVER_HOME" "$CLIENT_HOME"' EXIT
PASSWORDS="$SERVER_HOME/simple_auth_manager_passwords.json.generated"
for _ in $(seq 90); do
curl -sf "$URL/api/v2/monitor/health" > /dev/null && [ -s "$PASSWORDS" ]
&& break
sleep 1
done
PASSWORD=$(python3 -c 'import json, sys;
print(json.load(open(sys.argv[1]))["admin"])' "$PASSWORDS") || {
cat "$SERVER_HOME/api-server.log"
exit 1
}
export AIRFLOW_HOME="$CLIENT_HOME"
say 'airflow version # the server'
env "${SERVER_ENV[@]}" "$AIRFLOW" version 2> /dev/null
AIRFLOW_CLI_TOKEN=$("$CTL" auth token --api-url "$URL" --username admin
--password "$PASSWORD")
export AIRFLOW_CLI_TOKEN
say "airflowctl auth login --api-url $URL --skip-keyring #
AIRFLOW_CLI_TOKEN is set"
"$CTL" auth login --api-url "$URL" --skip-keyring
say "curl -s -H \"Authorization: Bearer \$AIRFLOW_CLI_TOKEN\"
$URL/api/v2/dagTags"
curl -s -H "Authorization: Bearer $AIRFLOW_CLI_TOKEN" "$URL/api/v2/dagTags"
echo
for output in json yaml table plain; do
say "airflowctl dags get-tags -o $output"
"$CTL" dags get-tags -o "$output"
echo "exit=$?"
done
```
</details>
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes, GitHub Copilot CLI (Claude Opus 5.5)
Generated-by: GitHub Copilot CLI (Claude Opus 5.5) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
