sureshanaparti opened a new pull request, #14231:
URL: https://github.com/apache/cloudstack/pull/14231
### Description
This PR adds support for OpenTelemetry distributed tracing instrumentation,
has the following changes
**Adds support to API layer**
All API requests get a traceId in LogContext (via ApiTraceFilter), captures
traceID from API. Async jobs store that context JSON in DB, restored before
execution. All logs show traceId automatically via %X{traceId}
the ApiTraceFilter that records a per request trace id on the log context
when a request runs inside an active OpenTelemetry span, it also records the
span trace id and span id on the log context, so log lines can be joined to the
distributed trace. The existing header or UUID traceid behaviour is unchanged.
No id is invented: when there is no valid span the keys are left unset and
render empty, and all keys are removed in the finally block. The filter lives
in the api module. LogContext gains the key constants and ThreadContext
write/remove support.
When the OTel Java agent is attached (via `-javaagent`), every API request
is traced with:
- **Per-command span naming**: Each trace is tagged with the API command
name (e.g. `listVirtualMachines`, `deployVirtualMachine`) via
`Span.current().updateName()` and a filterable `api.command` attribute, making
it possible to analyze latency per API command.
- **Automatic JDBC tracing**: The OTel agent auto-instruments all database
queries — every SELECT, INSERT, UPDATE shows up as a child span with table name
and duration. No code changes required for this.
- **Automatic HTTP client tracing**: Outbound HTTP calls (e.g. to hypervisor
agents) are auto-instrumented by the agent.
**All changes are no-ops without the OTel agent deployed.** The `@WithSpan`
annotation is ignored, `Span.current()` returns a no-op, and the
`opentelemetry-api` calls return immediately with zero overhead. The two added
dependencies (`opentelemetry-instrumentation-annotations` and
`opentelemetry-api`) are lightweight JARs (~50KB + ~200KB) with no transitive
dependencies.
Changes:
- `server/pom.xml`: Add `opentelemetry-instrumentation-annotations` (2.16.0)
and `opentelemetry-api` (1.51.0) dependencies
- `api/pom.xml`: Add `opentelemetry-instrumentation-annotations` (2.16.0)
dependency
- `ApiServer.java`: Add `@WithSpan` on `handleRequest()` with dynamic span
naming using the API command parameter and `api.command` span attribute
- `supervisord.conf`: Add `redirect_stderr=true` to cloudstack process so
OTel agent startup logs are visible in container logs
**Instrument cloudstack Agents and VM operations**
- Adds OpenTelemetry spans to the management server's agent command and VM
work job handlers so that hypervisor-bound traffic and VM operations can be
traced end-to-end.
Changes:
- **Outbound agent commands** (`AgentAttache.send`) are now wrapped in a
`CLIENT` span named `agent.out.<CommandName>`, tagged with the traffic type,
command name, host id, and an agent-call marker.
- **Inbound agent requests** (`AgentManagerImpl.processRequest`) are now
wrapped in a `SERVER` span named `agent.in.<CommandName>`, tagged with the same
set of attributes so the incoming side of a command can be correlated with the
outgoing side.
- **VM work jobs** (`VirtualMachineManagerImpl.handleVmWorkJob`) are now
wrapped in a span named after the work operation (e.g. `VmWorkStart`), tagged
with the operation, VM id, an op-root marker, and the resulting job status. The
VM operation, VM id, and traffic type are also propagated as OpenTelemetry
**baggage** for the duration of the job so downstream spans inherit that
context.
- A new `TracingLabels` utility class centralizes the span attribute /
baggage keys (and shared values like the `hypervisor` traffic type) so
instrumentation stays consistent across handlers.
- Adds the `opentelemetry-api` and
`opentelemetry-instrumentation-annotations` dependencies to
`engine/orchestration`.
This is purely additive instrumentation. When no OpenTelemetry agent/SDK is
attached, the API is a no-op and behavior is unchanged. `handleVmWorkJob`
guards against a null result before recording the job-status attribute.
**Read the trace and span threadcontext key names from the environment**
The ThreadContext key names used for the OpenTelemetry trace and span ids
were hardcoded in LogContext. This reads them from the environment instead, so
the key names are deployment specific rather than baked into core:
- `CLOUDSTACK_TRACE_ID_MDC_KEY`, default `otel_trace_id`
- `CLOUDSTACK_SPAN_ID_MDC_KEY`, default `otel_span_id`
The default applies whenever the property is absent or blank, so a
deployment that configures nothing still gets working, self describing key
names. A deployment that needs a particular field name in its log pipeline sets
the properties, and nothing in core has to know about it.
Changes:
Four files: the constants and property lookup in LogContext, the two usages
plus a javadoc line in TraceContextMdcWrapper, the assertions in
TraceContextMdcWrapperTest, and one comment in ServerDaemon. No behaviour
changes beyond where the key names come from.
<!--- Describe your changes in DETAIL - And how has behaviour functionally
changed. -->
<!-- For new features, provide link to FS, dev ML discussion etc. -->
<!-- In case of bug fix, the expected and actual behaviours, steps to
reproduce. -->
<!-- When "Fixes: #<id>" is specified, the issue/PR will automatically be
closed when this PR gets merged -->
<!-- For addressing multiple issues/PRs, use multiple "Fixes: #<id>" -->
<!-- Fixes: # -->
<!---
*******************************************************************************
-->
<!--- NOTE: AUTOMATION USES THE DESCRIPTIONS TO SET LABELS AND PRODUCE
DOCUMENTATION. -->
<!--- PLEASE PUT AN 'X' in only **ONE** box -->
<!---
*******************************************************************************
-->
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] Build/CI
- [ ] Test (unit or integration test code)
### Feature/Enhancement Scale or Bug Severity
#### Feature/Enhancement Scale
- [x] Major
- [ ] Minor
#### Bug Severity
- [ ] BLOCKER
- [ ] Critical
- [ ] Major
- [ ] Minor
- [ ] Trivial
### Screenshots (if appropriate):
### How Has This Been Tested?
- Run the management server with the OpenTelemetry Java agent attached:
exercised agent commands and VM lifecycle operations (start/stop/migrate) and
confirmed spans appear in the tracing backend with the expected names
(agent.out.*, agent.in.*, VmWork*) and attributes, and that the
inbound/outbound agent spans and downstream VM-work spans correlate via the
propagated baggage. With no agent attached, the OpenTelemetry API is a no-op
and behavior is unchanged.
- Confirmed in Jaeger with traces and spans. Verified traces flow.
- Confirmed API command names appear correctly in span names (e.g.
ApiServer.handleRequest listCapabilities).
- Confirmed api.command attribute is filterable in Tempo via
{span.api.command="listCapabilities"}
- Confirmed JDBC auto-instrumentation traces all DB queries as child spans
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to
-->
#### How did you try to break this feature and the system with this change?
<!-- see how your change affects other areas of the code, etc. -->
<!-- Please read the
[CONTRIBUTING](https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md)
document -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
