hanxdatadog opened a new issue, #76104:
URL: https://github.com/apache/airflow/issues/76104
**Is your feature request related to a problem?**
In multi-team deployments, Dag tags like `team:payments` are the natural way
to say who owns a Dag. The metrics side already supports this via `[metrics]
dag_tags_in_metrics` (#68568), and #37901 asks for the same for Sentry alert
routing. Task logs have no equivalent, so log-based routing and filtering (e.g.
"send this team its failed-task logs") can't use Dag tags.
The workaround today is a listener that calls
`structlog.contextvars.bind_contextvars(...)` in `on_task_instance_running`. We
run this in production and it mostly works, but it has two gaps that can't be
fixed from a listener:
1. **Early log lines have no tags.** `on_task_instance_running` is called at
the end of `_prepare()`. Anything logged earlier has no tags: Dag parsing,
`render_templates()`, map-index rendering and inlet/outlet validation. That
includes failures. If template rendering raises, `_prepare()` exits before the
listener runs, so the traceback for that failed task can't be attributed to its
owner. Lines after the terminal hooks (where the listener has to unbind) are
also untagged.
2. **Cleanup is easy to get wrong.** A correct listener has to unbind in
`on_task_instance_success`, `on_task_instance_failed`,
`on_task_instance_skipped` and `before_stopping`, plus defensively at the next
`on_task_instance_running`. Otherwise a deferred or rescheduled attempt in a
reused process can carry stale tags into the next attempt. The listener API
also doesn't document that bound contextvars reach task logs; it works only
because the task logging chain includes `merge_contextvars`.
**Proposed solution:**
Bind Dag tags in `task_runner.startup()` right after `parse()`, next to the
existing `ti_id` / `dag_id` / `task_id` / ... binding. The tags then cover the
whole attempt with no cleanup. Gate it behind an opt-in option, mirroring
`dag_tags_in_metrics`.
Put all tags under **one** key rather than one top-level key per tag, so a
tag like `level:x`, `event:x` or `logger:x` can't overwrite standard log fields:
```json
{"event": "...", "level": "info", "dag_id": "...", "dag_tags": {"team":
"payments", "tier": "1"}}
```
`key:value` tags would split the same way as `dag_tags_in_metrics`
(`build_dag_metric_tags()`); plain tags map to an empty value.
**Alternative solutions you considered:**
- **Listener plugin (status quo):** has the gaps described above.
**Who would benefit?**
Operators of multi-team deployments who route, filter or attribute task logs
by owner, environment or criticality.
**Additional context:**
- Draft implementation: #71793.
- Related: #68568 (metrics counterpart), #37901 (Dag tags for Sentry
routing).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
