songzhendong commented on PR #13891:
URL: https://github.com/apache/skywalking/pull/13891#issuecomment-4655282594
Thank you for your suggestions on the solution.
SWIP-7 was drafted a long time ago. This monitoring feature was originally
developed for Airflow 2.x, which exposes metrics in the `airflow.dataset.*`
format. By the time we finished development and updated the deployment
documents, we had set Airflow 3.x as the targeted version. However, we failed
to revalidate the MAL rules and E2E test data against the new release, which
resulted in this version mismatch.
---
### Changes in this commit
#### 1. Align version and MAL rules for Airflow 3.x
- **MAL rules**: Asset metric collection now uses `airflow_asset_*` (Airflow
3.x native OTel names) instead of `airflow.dataset.*` (Airflow 2.x).
- **Documentation**: `backend-airflow-monitoring.md` and `SWIP-7.md` now
clearly state that this feature targets Airflow 3.x. Metrics prefixed with
`airflow.dataset.*` from Airflow 2.x are no longer collected.
- **Test data**: All mocked OTLP replay fixtures use `airflow.asset.*`. The
cluster E2E image is pinned to Airflow 3.0.6.
#### 2. Additional improvements
- **E2E layout**: Test cases are split into `mock/` and `cluster/`,
following the layout used by other monitoring modules (e.g. Flink):
- `mock/`: 29 checks — full OTLP JSON replay covering all 27 SWIP-7
metrics.
- `cluster/`: 16 checks — native scheduler/triggerer smoke tests on
Airflow 3.0.6.
- **CI / local parity**: Custom shell verifiers (`verify-mock-e2e.sh`,
`verify-cluster-e2e.sh`, etc.) are removed. Validation now uses the standard
`e2e run` flow in both local environments and CI
(`.github/workflows/skywalking.yaml`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
