tailg8nj opened a new issue, #70388:
URL: https://github.com/apache/airflow/issues/70388
### Description
## Background
In attempting to leverage the auto-instrumentation mechanism with a vendored
distribution , I have run into a few issues. I believe with a few small tweaks
this could be much polished and remove the need for workarounds.
## Setup
Auto instrumentation has two ways of being invoked, one using the command
line wrapper `opentelemetry-instrument` and the other explicitly invoking it in
code. Some of the issues identified later in this document force the latter
approach.
Placing the following at the top of `airflow_local_settings.py` achieves
autoinstrumentation:
```python
auto_instrumentation.initialize()
from airflow._shared.observability.traces import
OverrideableRandomIdGenerator
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.trace import get_tracer_provider
if isinstance((tracer_provider := get_tracer_provider()), TracerProvider):
tracer_provider.id_generator = OverrideableRandomIdGenerator()
```
## Issue 1: OverrideableRandomIdGenerator
*Change Difficulty: Easy*
This class was introduced in #63839. My understanding is this a key piece in
the task span propagation strategy, but also looks alot like an anti-pattern:
creating a span, just to throw it away, and re-use it's IDs later. This is not
a blocker. See my comment requesting a more detailed explanation
https://github.com/apache/airflow/pull/63839#discussion_r3544019341.
The workaround is to explicitly override any Distro's ID Generator, e.g.
```python
tracer_provider.id_generator = OverrideableRandomIdGenerator()
```
This class is marked private.
OpenTelemetry offers an environment variable to choose your ID generator, so
long as the class is a defined entry point.
Could it be exposed using a Python Entry Point so that others could enforce
it via ENV VAR such that I wouldn't have to reference it directly?
* [Upstream ID Generator
Creation](https://github.com/open-telemetry/opentelemetry-python/blob/634cec5f2a2fecb40cb9d8216888c7b8865b845a/opentelemetry-sdk/src/opentelemetry/sdk/_configuration/__init__.py#L627)
* [Upstream ID Generator Environment
Variable](https://github.com/open-telemetry/opentelemetry-python/blob/634cec5f2a2fecb40cb9d8216888c7b8865b845a/opentelemetry-api/src/opentelemetry/environment_variables/__init__.py#L49)
* [Upstream ID Generator Entry
Point](https://github.com/open-telemetry/opentelemetry-python/blob/634cec5f2a2fecb40cb9d8216888c7b8865b845a/opentelemetry-sdk/pyproject.toml#L43-L44)
* [Lack of Entry Point in
Airflow](https://github.com/apache/airflow/blob/a050861fc59193347a60d6ab306a5e0a03c719a7/shared/observability/pyproject.toml)
## Issue 2: Hardcoded Propagation logic
*Change Difficulty: Easy*
OpenTelemetry vendors may choose to leverage TraceState to propagate
implementation-specific details, see definition in
[SpanContext](https://opentelemetry.io/docs/concepts/signals/traces/#span-context).
Coupled with the fact that [TraceState is
immutable](https://opentelemetry.io/docs/specs/otel/trace/api/#tracestate),
vendors may also supply custom propagators (not just because of a different
carrier format):
> Please note, since SpanContext is immutable, it is not possible to update
SpanContext with a new TraceState. Such changes then make sense only right
before SpanContext propagation or telemetry data exporting. In both cases,
Propagators and SpanExporters may create a modified TraceState copy before
serializing it to the wire.
The current strategy at the Airflow boundary was introduced in #69633. See
callout to [hardcoding the
propagator](https://github.com/apache/airflow/pull/69633/changes#r3640039700).
This issue also holds true for Baggage.
Workarounds may exist for this depending on the situation.
## Issue 3: DAG spans and cross-task propagation
*Change Difficulty: Moderate*
A incongruency exists between the DAG / trace waterfall and task dependency.
How do we intend to offer OpenTelemetry vendors a way to propagate
information across tasks via context_carriers?
Distros may leverage TraceState or Baggage.
The task span is created from the dag run context and not from the previous
task context
https://github.com/apache/airflow/blob/83edc67794b0e33233c6d4cdca8300e66b345939/airflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py#L568
Because the dag_run.context_carrier is not exposed at the task level, there
is no way OpenTelemetry native way to inject or propagate info to downstream
tasks.
Ideally a task should be able to inject information into a context_carrier
and have it be extracted for dependent tasks.
Workarounds using XCom exist, and maybe this is the correct place for it
seeing as it is the intended cross-task communication medium.
See my comment regarding the [resolution of context_carrier vs
parent_context_carrier](https://github.com/apache/airflow/issues/63281#issuecomment-4916089591).
### Use case/motivation
There's been significant progress in integration OpenTelemetry to Airflow.
As it stands right now, OpenTelemetry configuration is baked into the
`airflow.cfg` configuration flow. However, OpenTelemetry also provides a way to
auto-instrument Python applications as well as for vendors to ship their own
bundled set of configurations and key SDK components, see: [OpenTelemetry
distribution](https://opentelemetry.io/docs/languages/python/distro/),
[OpenTelemetry
autoinstrumentation](https://opentelemetry.io/docs/zero-code/python/).
### Related issues
_No response_
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
