wu-sheng commented on PR #13891:
URL: https://github.com/apache/skywalking/pull/13891#issuecomment-4631482030
**`dataset.*` vs `asset.*` — the asset metrics only resolve on Airflow 2.x;
on 3.x they'll be empty**
The asset rules read the Airflow **2.x** metric names, but the setup doc
targets 3.x ("Example environment variables for Airflow 3.x"). I checked the
Airflow source for both lines (`v2-10-stable` and `main`):
| SkyWalking metric | Reads source | Airflow 2.10 emits | Airflow 3.x (main)
emits |
|---|---|---|---|
| `asset_updates` | `airflow_dataset_updates` |
`Stats.incr("dataset.updates")` — `datasets/manager.py:149` |
`stats.incr("asset.updates")` — `assets/manager.py:400` |
| `asset_triggered_dagruns` | `airflow_dataset_triggered_dagruns` |
`Stats.incr("dataset.triggered_dagruns")` — `scheduler_job_runner.py:1494` |
`stats.incr("asset.triggered_dagruns")` — `scheduler_job_runner.py:2454` |
| `asset_orphaned` | `airflow_dataset_orphaned` |
`Stats.gauge("dataset.orphaned")` — `scheduler_job_runner.py:2200` |
`stats.gauge("asset.orphaned")` — `scheduler_job_runner.py:3378` |
This is the AIP-74/75 **Dataset → Asset** rename. In Airflow 3.x there are
no `dataset.*` Stats calls left, so OAP receives `airflow_asset_*` and the
current rules (filtering `airflow_dataset_*`) collect nothing → the three
`asset_*` panels stay empty on 3.x. The other 25 metrics (scheduler / executor
/ pool / triggerer / triggers / dag_processing) are byte-identical across 2.x
and 3.x, so only these three are affected. (The e2e cluster pins 2.10.5, which
is why it passes today.)
Not a blocker — just a version-targeting mismatch. Options:
1. **Doc-only:** state that the asset/dataset metrics are collected for
Airflow **2.x** (`dataset.*`) and that 3.x `asset.*` support is pending.
2. **Support both:** add parallel rules reading `airflow_asset_*` alongside
the `airflow_dataset_*` ones (the SkyWalking-side metric is already named
`asset_*`, so it lines up) — then the layer works on 2.x and 3.x.
Either way, worth making the docs + e2e target consistent. Happy to push a
suggestion for whichever direction you prefer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
