xBis7 commented on PR #63757:
URL: https://github.com/apache/airflow/pull/63757#issuecomment-4070675002
@potiuk @ashb @dstandish The new script should fail in the CI but I would
like to get some feedback first. Once you think that the script and the failing
output are looking good, then I'll push the fixes to the YAML file.
This is the output that I get when running it locally
```zsh
> prek run check-metrics-synced-with-registry --all-files
Check that metrics in the codebase are in sync with the metrics registry
YAML file........................Failed
- hook id: check-metrics-synced-with-registry
- exit code: 1
Found 1 violation(s).
-> 1 metric(s) found in the code but missing from the registry YAML:
providers/openlineage/src/airflow/providers/openlineage/plugins/adapter.py line
184: ol.emit.failed (incr) [Stats]
Add them to the registry before using them in the code.
Found 3 violation(s).
-> 3 metric(s) found in the code but missing from the registry YAML:
providers/openlineage/src/airflow/providers/openlineage/plugins/listener.py
line 239: ol.event.size.{event_type}.{operator_name} (dynamic) (gauge) [Stats]
providers/openlineage/src/airflow/providers/openlineage/plugins/listener.py
line 371: ol.event.size.{event_type}.{operator_name} (dynamic) (gauge) [Stats]
providers/openlineage/src/airflow/providers/openlineage/plugins/listener.py
line 519: ol.event.size.{event_type}.{operator_name} (dynamic) (gauge) [Stats]
providers/openlineage/src/airflow/providers/openlineage/plugins/listener.py
line 642: ol.event.size.{event_type}.{operator_name} (dynamic) (gauge) [Stats]
providers/openlineage/src/airflow/providers/openlineage/plugins/listener.py
line 747: ol.event.size.{event_type}.{operator_name} (dynamic) (gauge) [Stats]
providers/openlineage/src/airflow/providers/openlineage/plugins/listener.py
line 207: ol.extract.{event_type}.{operator_name} (dynamic) (timer) [Stats]
providers/openlineage/src/airflow/providers/openlineage/plugins/listener.py
line 340: ol.extract.{event_type}.{operator_name} (dynamic) (timer) [Stats]
providers/openlineage/src/airflow/providers/openlineage/plugins/listener.py
line 487: ol.extract.{event_type}.{operator_name} (dynamic) (timer) [Stats]
providers/openlineage/src/airflow/providers/openlineage/plugins/listener.py
line 611: ol.extract.{event_type}.{operator_name} (dynamic) (timer) [Stats]
task-sdk/src/airflow/sdk/serde/__init__.py line 377:
serde.load_serializers (timer) [Stats]
Add them to the registry before using them in the code.
Found 2 violation(s).
-> 1 metric(s) found in the code but missing from the registry YAML:
providers/amazon/src/airflow/providers/amazon/aws/executors/aws_lambda/lambda_executor.py
line 468: lambda_executor.adopt_task_instances.duration (timer) [Stats]
Add them to the registry before using them in the code.
-> 1 metric(s) found in code with a type that doesn't match the
registry:
airflow-core/src/airflow/jobs/scheduler_job_runner.py line 2966:
asset.orphaned (gauge) [Stats] -- code type: gauge, registry type: counter
Fix the type mismatch in either the code or the registry.
Found 4 violation(s).
-> 4 metric(s) found in the code but missing from the registry YAML:
providers/amazon/src/airflow/providers/amazon/aws/executors/batch/batch_executor.py
line 493: batch_executor.adopt_task_instances.duration (timer) [Stats]
providers/edge3/src/airflow/providers/edge3/executors/edge_executor.py line
284: edge_executor.sync.duration (timer) [Stats]
providers/edge3/src/airflow/providers/edge3/executors/edge_executor.py line
199: edge_worker.ti.finish (incr) [Stats]
providers/edge3/src/airflow/providers/edge3/executors/edge_executor.py line
195: edge_worker.ti.finish.{queue}.{FAILED}.{dag_id}.{task_id} (dynamic) (incr)
[Stats]
Add them to the registry before using them in the code.
Found 5 violation(s).
-> 5 metric(s) found in the code but missing from the registry YAML:
airflow-core/src/airflow/dag_processing/manager.py line 1276:
dag_processing.callback_only_count (incr) [Stats]
providers/edge3/src/airflow/providers/edge3/worker_api/routes/jobs.py line 152:
edge_worker.ti.finish (incr) [DualStatsManager]
providers/edge3/src/airflow/providers/edge3/worker_api/routes/jobs.py line 161:
edge_worker.ti.finish (incr) [Stats]
providers/edge3/src/airflow/providers/edge3/worker_api/routes/jobs.py line 157:
edge_worker.ti.finish.{queue}.{state}.{dag_id}.{task_id} (dynamic) (incr)
[Stats]
providers/edge3/src/airflow/providers/edge3/worker_api/routes/jobs.py line 95:
edge_worker.ti.start (incr) [DualStatsManager]
providers/edge3/src/airflow/providers/edge3/worker_api/routes/jobs.py line 98:
edge_worker.ti.start (incr) [Stats]
providers/edge3/src/airflow/providers/edge3/worker_api/routes/jobs.py line 97:
edge_worker.ti.start.{queue}.{dag_id}.{task_id} (dynamic) (incr) [Stats]
Add them to the registry before using them in the code.
Found 1 violation(s).
-> 1 metric(s) found in the code but missing from the registry YAML:
airflow-core/src/airflow/serialization/definitions/dag.py line
662: deadline_alerts.deadline_created (incr) [Stats]
Add them to the registry before using them in the code.
Found 2 violation(s).
-> 2 metric(s) found in the code but missing from the registry YAML:
airflow-core/src/airflow/models/deadline.py line 263:
deadline_alerts.deadline_missed (incr) [Stats]
airflow-core/src/airflow/models/deadline.py line 197:
deadline_alerts.deadline_not_missed (incr) [Stats]
Add them to the registry before using them in the code.
Found 1 violation(s).
-> 1 metric(s) found in the code but missing from the registry YAML:
providers/amazon/src/airflow/providers/amazon/aws/executors/ecs/ecs_executor.py
line 606: ecs_executor.adopt_task_instances.duration (timer) [Stats]
Add them to the registry before using them in the code.
Found 1 violation(s).
-> 1 metric(s) found in the code but missing from the registry YAML:
task-sdk/src/airflow/sdk/io/fs.py line 57:
airflow.io.load_filesystems (timer) [Stats]
Add them to the registry before using them in the code.
Found 2 violation(s).
-> 2 metric(s) found in code with a type that doesn't match the
registry:
providers/edge3/src/airflow/providers/edge3/worker_api/routes/worker.py line
221: edge_worker.heartbeat_count (incr) [DualStatsManager] -- code type:
counter, registry type: gauge
providers/edge3/src/airflow/providers/edge3/worker_api/routes/worker.py line
230: edge_worker.heartbeat_count (incr) [Stats] -- code type: counter, registry
type: gauge
providers/edge3/src/airflow/providers/edge3/worker_api/routes/worker.py line
229: edge_worker.heartbeat_count.{worker_name} (incr) [Stats] -- code type:
counter, registry type: gauge
Fix the type mismatch in either the code or the registry.
```
Unfortunately, the coloring is lost when pasting the text.
<img width="2834" height="2142" alt="image"
src="https://github.com/user-attachments/assets/b99af58b-60e6-4168-a4ec-bc236c11a87e";
/>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
