Andrey Yarovoy created HDDS-17350:
-------------------------------------
Summary: Ozone client builds the OpenTelemetry SDK at start-up
even when tracing is disabled
Key: HDDS-17350
URL: https://issues.apache.org/jira/browse/HDDS-17350
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Client
Reporter: Andrey Yarovoy
*Problem*
`RpcClient` calls `TracingUtil.initTracing("client", conf)` when it is created.
With tracing disabled
(`ozone.tracing.enabled=false`, the default), `TracingUtil.initialize()` only
skips the setup when
`ozone.tracing.client.application-aware` is also false. But that option
defaults to true (HDDS-13803). Unless the
application has registered its own `GlobalOpenTelemetry`, every client
therefore runs `initOzoneSdk()` →
`buildSdkTracerProvider()`. That means:
- parsing the sampler and span-sampling config (the "Sampling Trace Config" /
"Sampling Span Config" log lines);
- building an `OtlpGrpcSpanExporter` and a `BatchSpanProcessor`, which starts a
background export thread;
- building the `SdkTracerProvider` and `OpenTelemetrySdk`, and registering it
as `GlobalOpenTelemetry`.
In application-aware mode none of this is used unless a span is already active.
Spans are created only when
`Span.current()` is valid (`shouldCreateTracingProxy`), and a client never
starts a root span
(`canStartSpanWithoutParent()` returns `tracingEnabled`). For the usual client
process with no tracing of its own,
the SDK, exporter and thread are never used.
The cost shows up wherever many short-lived client JVMs start: YARN
`ContainerLocalizer` JVMs, MapReduce tasks, CLI
commands. In a fresh JVM on a secure compose cluster, timed with step markers
and DEBUG logs, `FileSystem.get()` for
`ofs://` takes about 600 ms. About **80 ms** of that runs from the first
`TracingUtil` line to "Initialized tracing
service: client (enabled=false, applicationAware=true)", including loading the
OpenTelemetry SDK and OTLP exporter
classes. A MapReduce job starts about four such cold clients one after another
on its critical path, so this is
roughly 0.3 s per job.
*Proposed change*
Build the SDK lazily in application-aware mode:
1. In `initialize()`, when tracing is disabled and application-aware is on:
- if the application registered a real `GlobalOpenTelemetry`, adopt it, as
today; this is cheap;
- otherwise keep the no-op tracer and record that an SDK may be needed, without
building it.
2. On the first span request that has a valid parent, build Ozone's SDK once,
under the existing lock, and
continue as today. The parent can come from `Span.current()` in
`shouldCreateTracingProxy` and the child-span
paths, or from a non-empty propagated context in `importSpan`.
`hasUsableTracer()` must treat "SDK pending" as
usable, so those checks still pass.
3. Nothing changes when `ozone.tracing.enabled=true`: the SDK is built eagerly,
so root spans can start at once.
*Compatibility*
There's no change in which spans are created or exported. The differences:
- the export thread and exporter only exist once a traced call happens;
- in application-aware mode, Ozone registers its SDK as `GlobalOpenTelemetry`
at the first traced call instead of at
client creation;
- a co-resident library that reads `GlobalOpenTelemetry` before that sees the
default no-op instance, as it would
in a process with tracing fully disabled. This is worth calling out in the docs.
*Tests*
- With tracing disabled and application-aware on: after client creation no
`SdkTracerProvider` or
`BatchSpanProcessor` exists, and `GlobalOpenTelemetry` isn't set.
- The same setup with an active application span around an Ozone call: the SDK
is built once, and the call produces
a child span.
- An application-registered `GlobalOpenTelemetry` is still adopted without
building Ozone's SDK.
- `ozone.tracing.enabled=true`: unchanged, eager SDK.
- Optional: the cold-client probe shows `FileSystem.get()` about 80 ms shorter.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
