This is an automated email from the ASF dual-hosted git repository.
chenhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 6b76544433b [improve] [pip] PIP-355: Enhancing Broker-Level Metrics
for Pulsar (#22778)
6b76544433b is described below
commit 6b76544433b43185e1990a919a6d5a5b8cf236d6
Author: Hang Chen <[email protected]>
AuthorDate: Tue Jun 11 15:56:43 2024 +0800
[improve] [pip] PIP-355: Enhancing Broker-Level Metrics for Pulsar (#22778)
Co-authored-by: Dragos Misca <[email protected]>
---
pip/pip-355.md | 36 ++++++++++++++++++++++++++++++++++++
1 file changed, 36 insertions(+)
diff --git a/pip/pip-355.md b/pip/pip-355.md
new file mode 100644
index 00000000000..cb0e41faefd
--- /dev/null
+++ b/pip/pip-355.md
@@ -0,0 +1,36 @@
+# PIP-355: Enhancing Broker-Level Metrics for Pulsar
+
+# Background Knowledge
+Pulsar provides broker-level, namespace-level, and topic-level metrics to
monitor and analyze the behavior of the Pulsar service. These metrics are
accessible through the Prometheus metrics endpoint. Detailed explanations of
all metrics can be found on the Pulsar website: [Pulsar Metrics
Reference](https://pulsar.apache.org/docs/3.2.x/reference-metrics/)
+
+# Motivation
+Within Pulsar's current metrics framework, the `pulsar_out_bytes_total` metric
is utilized to expose the total bytes dispatched by the broker to consumers.
However, there are notable limitations and challenges associated with this
metric:
+- Inclusion of system subscriptions in the total bytes out, alongside user
subscriptions, complicates accurate calculation of user-specific data.
+- The granularity of the metric (namespace-level vs. topic-subscription level)
impacts the scalability and resource consumption when calculating cluster-level
total out bytes.
+
+# Goals
+This proposal aims to address the following objectives:
+- Simplify the process of calculating cluster-level total out bytes.
+- Enable the calculation of total out bytes dispatched to system subscriptions.
+
+# High-Level Design
+To achieve the outlined goals, the proposal introduces two new broker-level
metrics:
+- `pulsar_broker_out_bytes_total{system_subscription="true|false"}`:
Represents the total out bytes dispatched by the broker to consumers. The label
`system_subscription="false"` represents total traffic dispatched to user
subscriptions, while `system_subscription="true"` represents total traffic
dispatched to system cursors and cursor names added by
`additionalSystemCursorNames` introduced in
[PIP-349](https://github.com/apache/pulsar/pull/22651).
+- `pulsar_broker_in_bytes_total{system_topic="true|false"}`: Tracks the total
in bytes sent by producers to the broker. The label `system_topic="false"`
represents total traffic from user topics, while `system_topic="true"`
represents total traffic from system topics.
+
+# Detailed Design
+The implementation involves the introduction of the following broker-level
metrics:
+- `pulsar_broker_out_bytes_total{system_subscription="true|false"}`:
Aggregates the total out bytes from all topics, presented as a broker-level
metric.
+- `pulsar_broker_in_bytes_total{system_topic="true|false"}`: Calculation of
total in bytes across all topics.
+
+# Metrics
+The proposal includes the addition of two new broker-level metrics:
+- `pulsar_broker_out_bytes_total{system_subscription="true|false"}`
+- `pulsar_broker_in_bytes_total{system_topic="true|false"}`
+
+# Backward & Forward Compatibility
+The proposed changes ensure full compatibility with existing systems and pave
the way for seamless integration with future enhancements.
+
+# Links
+- Mailing List discussion thread:
https://lists.apache.org/thread/n3vvh6pso9ml7sg3qpww870om5vcfnpv
+- Mailing List voting thread:
https://lists.apache.org/thread/h4rjcv77wppz96gc31cpr3hw17v9jc4o
