dragosvictor commented on code in PR #22010:
URL: https://github.com/apache/pulsar/pull/22010#discussion_r1484884025
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java:
##########
@@ -897,6 +902,7 @@ public void start() throws PulsarServerException {
}
this.metricsGenerator = new MetricsGenerator(this);
+ this.openTelemetry = new PulsarBrokerOpenTelemetry(config);
Review Comment:
High-level details about the no-op implementation can be found
[here](https://opentelemetry.io/docs/specs/otel/metrics/noop/).
For a more in-depth view, the auto-configured SDK instantiates an empty
`OpenTelemetrySdk` if telemetry is disabled
([ref](https://github.com/open-telemetry/opentelemetry-java/blob/ee6c9867d736af7359003e0b2f61ebf082e6fe3e/sdk-extensions/autoconfigure/src/main/java/io/opentelemetry/sdk/autoconfigure/AutoConfiguredOpenTelemetrySdkBuilder.java#L402)).
Since this object does not have any metric readers configured, it will use the
static no-op `MeterProvider`
([ref](https://github.com/open-telemetry/opentelemetry-java/blob/f421ef1e73906f6332f4a5ab39698b00efb4525c/sdk/metrics/src/main/java/io/opentelemetry/sdk/metrics/SdkMeterProvider.java#L96)),
which in turn relies on the no-op `MeterBuilder`
([ref](https://github.com/open-telemetry/opentelemetry-java/blob/695ed5350b2ef218c38087a627549f4de7b86c22/api/all/src/main/java/io/opentelemetry/api/metrics/DefaultMeterProvider.java#L16))
and finally the no-op `Meter` object
([ref](https://github.com/open-telemetry/opentelemetry-java/blob/695ed5
350b2ef218c38087a627549f4de7b86c22/api/all/src/main/java/io/opentelemetry/api/metrics/DefaultMeter.java#L19)).
In effect, the little memory resources that it uses even when disabled are a
handful of null fields and empty collections in `OpenTelemetrySdk` and its
children providers.
Regarding actual runtime usage, both CPU and memory, it will depend on how
we leverage the library. PIP-264 describes the overall plan:
- High-cardinality metrics shall be fetched via a batch callback, on top of
'asynchronous' meters. The benefit here is that we can iterate over an entire
collection (say, topics) just once and record each object's metrics in one
shot. This also lets us instantiate the respective `Attribute` array,
presumably different for each entity, just one time, thereby reducing memory
allocations. The associated cost is the meter itself, which must be built, but
it would reference the same underlying object when the library is disabled.
Since the callback is expected to perform the iteration, we don't need a meter
for each object, just one per object "class" (say, `TopicStats`); for instance,
if we assume that topics have 100 metrics associated, we will add 100 objects
to memory, independent of how many topics the broker is hosting. Each metric is
different, but our goal is to leverage these callbacks to pull the numbers from
existing data sources (`AtomicInteger`, `LongAdder`, etc.). Even be
tter, the callback does nothing
([ref](https://github.com/open-telemetry/opentelemetry-java/blob/695ed5350b2ef218c38087a627549f4de7b86c22/api/all/src/main/java/io/opentelemetry/api/metrics/DefaultMeter.java#L64))
if the library is disabled, so the compute overhead is effectively zero. This
should be our most common usage pattern.
- Histogram meters are 'synchronous' in OpenTelemetry jargon. This means we
must create an object for each entity type we want to monitor. If we want
namespace-level histograms for a metric, we will have to instantiate such a
histogram for each namespace. Because of this, the consensus in the PIP was to
eliminate such metrics from high-cardinality objects
([ref](https://github.com/apache/pulsar/blob/master/pip/pip-264.md#moving-topic-level-histograms-to-namespace-and-broker-level-only)).
This keeps memory usage under control. These histograms do nothing when the
library is disabled, so the CPU usage is practically zero again.
I am happy to explain a bit more if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
