[
https://issues.apache.org/jira/browse/HDDS-17581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
rajesh updated HDDS-17581:
--------------------------
Description:
h3. Background
As part of HDDS-13679, OM (and later SCM and datanodes) will create their
internal components as tracing proxies through their interfaces: for OM, the
request handler, client requests, the key, bucket and volume managers, and the
authorizer. Code inside request handling will also add attributes, events and
errors to the current span.
The tracing code in \{{hdds-common}} (\{{TracingUtil}}, \{{TraceAllMethod}})
doesn't support this yet:
* \{{createProxy(delegate, itf, conf)}} only creates a proxy when there is a
current span, so it can't wrap components created at service startup.
* Every proxy span is CLIENT unless the method has \{{@InternalSpanKind}},
which is wrong for in-process components.
* Proxy spans are named after the interface, not after the class doing the work.
* Exceptions from the delegate aren't recorded on the proxy span, and
\{{executeInSpan}} and \{{OzoneProtocolMessageDispatcher}} set only the ERROR
status, without the exception or its message.
* There are no null-safe helpers for adding attributes, events or exceptions to
the current span, and no test helper for asserting on finished spans.
This sub-task adds these building blocks in \{{hdds-common}} only. It doesn't
add new spans to any service; that is done in the follow-up OM, SCM and
datanode sub-tasks.
h3. Proposed changes
*Tracing proxies*
* \{{TraceAllMethod}} takes a \{{SpanKind}}. \{{@InternalSpanKind}} can be put
on a whole delegate class as well as on a method.
* New \{{createProxy(delegate, itf, spanKind)}} names spans \{{<delegate class
simple name>.<method>}}, for example \{{KeyManagerImpl.listStatus}}. It doesn't
need a current span, so it can wrap objects created at startup when
application-aware tracing is on. It returns the delegate itself when tracing is
neither enabled nor application-aware.
* \{{createProxy(delegate, itf, conf)}} now creates INTERNAL spans. New
\{{createClientProxy(delegate, itf, conf)}} creates CLIENT spans. The existing
RPC client proxies switch to \{{createClientProxy}}, so their span kind stays
CLIENT:
** the SCM block, container location, security and secret key clients, in
\{{HAUtils}} and \{{HddsServerUtil}};
** the OM client, in \{{RpcClient}}.
* Exceptions thrown by the delegate are recorded on the proxy span. Methods
missing from the delegate, and \{{@SkipTracing}} methods, are called on the
delegate directly instead of throwing \{{NoSuchMethodException}}.
*Span helpers in \{{TracingUtil}}*
* \{{createSpan(name)}} and \{{createSpan(name, SpanKind)}} create a child of
the current span.
* \{{recordException(span, t)}} and \{{recordExceptionInCurrentSpan(t)}} record
the exception event and set the ERROR status, with the exception message as the
description.
* \{{addAttributesToCurrentSpan(key, String|long|boolean)}} and
\{{addEventToCurrentSpan(name, key, value)}} do nothing when the current span
isn't recording, and skip null values.
* New \{{SpanAttributeKeys}} holds the attribute keys shared by services,
starting with \{{ozone.status}}.
*Error recording on existing spans*
* \{{executeInSpan}} and \{{OzoneProtocolMessageDispatcher#processRequest}} use
\{{recordException}}, so failed spans carry the exception event and message.
*Tests*
* New \{{TestSpanCollector}} installs an in-memory span processor and provides
assertions on span name, kind, parent, error status and recorded exceptions.
* New tests in \{{TestTracingUtil}}, \{{TestTraceAllMethod}},
\{{TestTracingAnnotations}}, \{{TestTracingInitModes}} and
\{{TestOzoneProtocolMessageDispatcher}}.
* Test tracer providers get a span processor: with OpenTelemetry 1.67, an
\{{SdkTracerProvider}} without a processor returns no-op spans.
h3. Compatibility
No proto, RPC, configuration or RocksDB changes. When tracing is disabled and
application-aware mode is off, proxies aren't created, so behaviour is
unchanged.
was:
h3. Why
This is the first, common-only step of adding OpenTelemetry tracing to OM (and
later SCM and datanodes). Upcoming PRs will:
* create OM components as tracing proxies through their interfaces, such as
the request handler, client requests, the key, volume and bucket managers, and
the authorizer;
* add attributes, events and error status from inside request handling.
For that we need a few things that {{hdds-common}} doesn't have yet:
* proxies that work at startup;
* proxies whose span kind matches what the delegate does: INTERNAL for
in-process components, CLIENT for RPC clients;
* exceptions recorded in a standard way;
* small null-safe helpers for the current span;
* a test helper to assert on collected spans.
This PR adds only these building blocks. It doesn't add new spans to any
service.
h3. What
*Span helpers in {{{}TracingUtil{}}}:*
* {{createSpan(name)}} and {{createSpan(name, SpanKind)}} create a span that
is a child of the current span.
* {{recordException(span, t)}} and {{recordExceptionInCurrentSpan(t)}} record
the exception event and set the ERROR status, using the exception message as
the description.
* {{addAttributesToCurrentSpan(key, String|long|boolean)}} and
{{addEventToCurrentSpan(name, key, value)}} do nothing when there is no
recording span, and skip null values.
* New {{SpanAttributeKeys}} holds the shared attribute keys, starting with
{{{}ozone.status{}}}.
*Tracing proxies:*
* {{TraceAllMethod}} takes a {{{}SpanKind{}}}. {{@InternalSpanKind}} can now
go on a whole delegate class as well as on a method.
* Exceptions thrown by the delegate are recorded on the proxy span. Methods
missing from the delegate, and {{@SkipTracing}} methods, are called directly
instead of throwing {{{}NoSuchMethodException{}}}.
* {{createProxy(delegate, itf, conf)}} now creates INTERNAL spans. The new
{{createClientProxy(delegate, itf, conf)}} creates CLIENT spans. The existing
RPC client proxies switch to {{{}createClientProxy{}}}, so their span kind
stays CLIENT:
** the SCM block, container location, security and secret key clients, in
{{HAUtils}} and {{{}HddsServerUtil{}}};
** the OM client in {{{}RpcClient{}}}.
* New {{createProxy(delegate, itf, spanPrefix, spanKind)}} names spans
{{{}<spanPrefix>.<method>{}}}, for example {{{}KeyManagerImpl.listStatus{}}}.
It doesn't need a current span, so it can wrap objects created at service
startup when application-aware tracing is on.
*Error recording on existing spans:*
* {{executeInSpan}} and {{OzoneProtocolMessageDispatcher#processRequest}} now
use {{{}recordException{}}}. Failed spans carry the exception event and the
error message, not only the ERROR status.
*Tests:*
* New {{TestSpanCollector}} installs an in-memory span processor and has
assertion helpers.
* New tests in {{{}TestTracingUtil{}}}, {{{}TestTraceAllMethod{}}},
{{{}TestTracingAnnotations{}}}, {{TestTracingInitModes}} and
{{{}TestOzoneProtocolMessageDispatcher{}}}.
* The test tracer providers get a span processor. With OpenTelemetry 1.67, an
{{SdkTracerProvider}} without a processor returns no-op spans.
> Add TracingUtil span helpers and span-kind aware tracing proxies
> ----------------------------------------------------------------
>
> Key: HDDS-17581
> URL: https://issues.apache.org/jira/browse/HDDS-17581
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Sravani Revuri
> Assignee: rajesh
> Priority: Major
>
> h3. Background
> As part of HDDS-13679, OM (and later SCM and datanodes) will create their
> internal components as tracing proxies through their interfaces: for OM, the
> request handler, client requests, the key, bucket and volume managers, and
> the authorizer. Code inside request handling will also add attributes, events
> and errors to the current span.
> The tracing code in \{{hdds-common}} (\{{TracingUtil}}, \{{TraceAllMethod}})
> doesn't support this yet:
> * \{{createProxy(delegate, itf, conf)}} only creates a proxy when there is a
> current span, so it can't wrap components created at service startup.
> * Every proxy span is CLIENT unless the method has \{{@InternalSpanKind}},
> which is wrong for in-process components.
> * Proxy spans are named after the interface, not after the class doing the
> work.
> * Exceptions from the delegate aren't recorded on the proxy span, and
> \{{executeInSpan}} and \{{OzoneProtocolMessageDispatcher}} set only the ERROR
> status, without the exception or its message.
> * There are no null-safe helpers for adding attributes, events or exceptions
> to the current span, and no test helper for asserting on finished spans.
> This sub-task adds these building blocks in \{{hdds-common}} only. It doesn't
> add new spans to any service; that is done in the follow-up OM, SCM and
> datanode sub-tasks.
> h3. Proposed changes
> *Tracing proxies*
> * \{{TraceAllMethod}} takes a \{{SpanKind}}. \{{@InternalSpanKind}} can be
> put on a whole delegate class as well as on a method.
> * New \{{createProxy(delegate, itf, spanKind)}} names spans \{{<delegate
> class simple name>.<method>}}, for example \{{KeyManagerImpl.listStatus}}. It
> doesn't need a current span, so it can wrap objects created at startup when
> application-aware tracing is on. It returns the delegate itself when tracing
> is neither enabled nor application-aware.
> * \{{createProxy(delegate, itf, conf)}} now creates INTERNAL spans. New
> \{{createClientProxy(delegate, itf, conf)}} creates CLIENT spans. The
> existing RPC client proxies switch to \{{createClientProxy}}, so their span
> kind stays CLIENT:
> ** the SCM block, container location, security and secret key clients, in
> \{{HAUtils}} and \{{HddsServerUtil}};
> ** the OM client, in \{{RpcClient}}.
> * Exceptions thrown by the delegate are recorded on the proxy span. Methods
> missing from the delegate, and \{{@SkipTracing}} methods, are called on the
> delegate directly instead of throwing \{{NoSuchMethodException}}.
> *Span helpers in \{{TracingUtil}}*
> * \{{createSpan(name)}} and \{{createSpan(name, SpanKind)}} create a child of
> the current span.
> * \{{recordException(span, t)}} and \{{recordExceptionInCurrentSpan(t)}}
> record the exception event and set the ERROR status, with the exception
> message as the description.
> * \{{addAttributesToCurrentSpan(key, String|long|boolean)}} and
> \{{addEventToCurrentSpan(name, key, value)}} do nothing when the current span
> isn't recording, and skip null values.
> * New \{{SpanAttributeKeys}} holds the attribute keys shared by services,
> starting with \{{ozone.status}}.
> *Error recording on existing spans*
> * \{{executeInSpan}} and \{{OzoneProtocolMessageDispatcher#processRequest}}
> use \{{recordException}}, so failed spans carry the exception event and
> message.
> *Tests*
> * New \{{TestSpanCollector}} installs an in-memory span processor and
> provides assertions on span name, kind, parent, error status and recorded
> exceptions.
> * New tests in \{{TestTracingUtil}}, \{{TestTraceAllMethod}},
> \{{TestTracingAnnotations}}, \{{TestTracingInitModes}} and
> \{{TestOzoneProtocolMessageDispatcher}}.
> * Test tracer providers get a span processor: with OpenTelemetry 1.67, an
> \{{SdkTracerProvider}} without a processor returns no-op spans.
> h3. Compatibility
> No proto, RPC, configuration or RocksDB changes. When tracing is disabled and
> application-aware mode is off, proxies aren't created, so behaviour is
> unchanged.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
