rajeshkumarchandolu opened a new pull request, #11607:
URL: https://github.com/apache/ozone/pull/11607
## What changes were proposed in this pull request?
Ozone services export OpenTelemetry spans, but their log lines don't
include the trace context, so it is hard to
go from a trace in Jaeger to the matching logs, or the other way round.
This change puts the current span's
`trace_id` and `span_id` into the slf4j MDC and prints them in the default
log patterns.
**Approach:** every `makeCurrent()` / `Context.wrap(...)` in OpenTelemetry
goes through
`ContextStorage.attach()`. OpenTelemetry chooses the `ContextStorage`
implementation through
`ServiceLoader<ContextStorageProvider>`. This PR registers
`OzoneContextStorageProvider` (hdds-common), which wraps
the default thread-local storage and keeps the MDC in sync with the
current span on attach and on scope close.
- No call-site changes. All existing spans, and any added later, get the
IDs in their log lines automatically.
- The MDC is derived from the current OpenTelemetry context rather than
saved and restored. This avoids
re-applying stale values that log4j's `InheritableThreadLocal` MDC
copies into child threads.
- The dist `log4j.properties` appenders now include `[trace=%X{trace_id}
span=%X{span_id}]`. The fields are
empty when tracing is off.
- The shaded ozonefs jar relocates the services file correctly through the
existing
`ServicesResourceTransformer`. If another provider is present on the
same classpath, OpenTelemetry falls back
to its default storage: tracing still works, but the log IDs are
missing.
**Known limitation (follow-up):** a thread created while a span is current
inherits that span's IDs through
log4j's MDC. If the thread never activates a span itself, it keeps logging
those IDs; for example, SCM
`EventQueue-*` threads created during a datanode `Register` RPC.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-13679
## How was this patch tested?
- **Unit tests:** new `TestOzoneContextStorageProvider` covers provider
registration, nested spans, invalid
spans, clearing a stale MDC value, and `Context.wrap` across threads.
The existing `TestTracingUtil` and
`TestTracingInitModes` pass, and so does `checkstyle.sh`.
- **Manual test:** a docker-compose cluster (`ozone` + `monitoring.yaml`)
with tracing enabled and Jaeger.
After `ozone sh key put`, the OM, datanode and SCM log lines carried
trace and span IDs that matched the
corresponding spans in Jaeger (e.g. OM `CreateKey` / `CommitKey`,
datanode `WriteChunk`). No
`ContextStorageProvider` warnings appeared in any service log.
##Build:
https://github.com/rajeshkumarchandolu/ozone/actions/runs/39101752286
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
