1fanwang opened a new issue, #66906: URL: https://github.com/apache/airflow/issues/66906
### Description AIP-59 added OpenTelemetry traces inside the scheduler, Dag processor, and task supervisor. The CLI entry points that drive those subsystems — `airflow tasks test`, `airflow dags trigger`, `airflow dags test`, `airflow backfill create` — are not wrapped in spans. When one of those commands is invoked from a wrapper that already carries trace context (a CI pipeline, a parent workflow, a debugger harness, a shell that received a `TRACEPARENT` env var), the inbound trace context dies at the CLI binary. The resulting task and DagRun spans show up as a *separate* trace; nothing stitches them back to the caller. The asymmetry is most visible when an external orchestrator triggers a Dag via the CLI and then tries to follow the work through to task completion: the orchestrator-side trace has no child spans, the Airflow-side trace has no parent. ### Use case/motivation Filing this from the LinkedIn Data Infrastructure Airflow / Orchestrator team. Scheduler and worker spans flow through our OTel pipeline today (AIP-59 coverage); CLI-issued operations don't. When something unexpected happens — a Dag run materialises in an unexpected state, a manual trigger from a CI job produces no follow-up runs, a task-test invocation from a developer's terminal misbehaves — we can't trace it back through OTel to the originating CLI command. Operators end up grepping logs by hand to reconstruct the sequence. The general shape is "every reasonable user of OTel hits the same wall when CLI commands sit in the middle of the trace chain." Wrapping the four CLI entry points in spans that honour W3C TraceContext closes that gap so: - An external orchestrator with a parent trace can invoke `airflow dags trigger ...` and see its trace ID stitch to Airflow's downstream task spans. - A CI pipeline invoking `airflow tasks test ...` for verification gets first-class span emission anchored to the pipeline's trace. - Debug harnesses that already export traces (custom test runners, breakpoint shells) can attach to the same trace without manual correlation. The default no-op behaviour stays unchanged when OTel isn't configured. ### Related issues Implementation in #66789. Builds on AIP-59 (existing tracer setup in `airflow_shared.observability.traces`). ### Are you willing to submit a PR? - [X] Yes I am willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
