This is an automated email from the ASF dual-hosted git repository.
zhaocong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 6b567e91797 [improve][pip] PIP-447: Customizable Prometheus Labels for
Topic Metrics (#25161)
6b567e91797 is described below
commit 6b567e91797c8b61ec410830bdec4d7f83227fe2
Author: Cong Zhao <[email protected]>
AuthorDate: Thu Feb 12 10:02:33 2026 +0800
[improve][pip] PIP-447: Customizable Prometheus Labels for Topic Metrics
(#25161)
---
pip/pip-447.md | 168 ++++++++++++++++++++++-----------------------------------
1 file changed, 64 insertions(+), 104 deletions(-)
diff --git a/pip/pip-447.md b/pip/pip-447.md
index d4cd9c15f62..0e3fbcd690f 100644
--- a/pip/pip-447.md
+++ b/pip/pip-447.md
@@ -2,7 +2,7 @@
# Abstract
-This PIP proposes a mechanism to add customizable Prometheus labels to Apache
Pulsar topic-level metrics. Administrators will define a set of allowed custom
metric label keys at the cluster or broker level. Users can then assign values
to these predefined keys for specific topics. These key-value pairs will be
exposed as Prometheus labels, enabling more granular metric filtering and
precise alerting, while managing metric cardinality through centralized key
governance.
+This PIP proposes a mechanism to selectively expose existing Pulsar Topic
Properties as Prometheus metrics labels. Instead of introducing a new mechanism
for custom labels, Administrators will define a set of allowed property keys at
both the broker level and namespace level. If a topic has a property matching
one of these allowed keys, its value will be automatically sanitized and
injected as a Prometheus label. This enables granular metric filtering and
alerting using the metadata user [...]
# Motivation
@@ -18,73 +18,48 @@ Operational Overhead: Increased effort in managing and
maintaining Prometheus al
Users require a native Pulsar mechanism to inject queryable, custom metadata
directly into topic metrics to improve alerting precision, simplify
dashboarding, and enhance overall observability.
-# Goals
-
-The proposed solution allows administrators to define a list of permissible
custom metric label keys (e.g., sla_tier, app_owner) in the broker
configuration.
-
-Users can then use pulsar-admin commands or the REST API to set string values
for these allowed keys on specific topics (e.g., sla_tier=gold).
+Users often already attach this metadata to topics using Topic Properties
(e.g., sla_tier=gold, owner=team_a). However, these properties are currently
invisible to the monitoring layer.
-These key-value pairs will be stored as part of the topic's metadata,
leveraging Pulsar's topic-level policy system.
-
-When topic metrics are generated for Prometheus, these custom key-value pairs
will be added as labels to the existing set of standard labels.
+# Goals
## In Scope
The primary goals of this proposal are to:
-Allow administrators to define a configurable set of allowed custom metric
label keys at the broker or cluster level.
-
-Enable users (or automated systems) to assign string values to these
predefined keys for individual topics.
-
-Expose these user-defined key-value pairs as additional Prometheus labels on
all topic-level metrics for the respective topics, provided
exposeTopicLevelMetricsInPrometheus is true.
-
-Provide robust control over Prometheus metric cardinality by restricting the
set of custom metric label keys and providing limits on their usage.
-
-Integrate this feature with Pulsar's existing topic-level policy framework,
utilizing system topics for policy propagation.
-
-## Out of Scope
+* Allow administrators to define a configurable list of allowed topic property
keys in the broker configuration.
-This PIP does not propose changes to topic-level properties. Topic-level
properties are distinct from topic-level custom metrics labels.
+* Allow administrators to define namespace-level allowed topic property keys
via admin API, providing finer-grained control per namespace.
-It does not aim to replace existing standard metric labels (cluster,
namespace, topic).
+* Update the Prometheus metrics generation logic to retrieve values from the
existing Topic Properties map and inject them as labels if they match the
allowed list (either broker-level or namespace-level).
-Support for complex data types as label values; only string values will be
supported.
+* Provide robust control over Prometheus metric cardinality by strictly
enforcing the allowed-list.
# High Level Design
-API and Data Structure Definition: Finalize the internal data structures for
storing custom metric labels within topic policies and the public contracts for
pulsar-admin commands and REST APIs.
+Configuration Implementation: Add parameters to broker.conf to define the
broker-level allow-list (e.g., allowedTopicPropertiesForMetrics). Add namespace
policy settings to enable namespace-level allow-list configuration via admin
API.
-Broker Configuration Implementation: Add the new configuration parameters to
broker.conf and implement the logic for brokers to read and use these settings.
+Metrics Generation Modification: Update the PrometheusMetricsServlet (or
equivalent exporter) to:
-Admin Client and REST API Implementation: Develop the new pulsar-admin topics
subcommands and their corresponding REST API endpoints in the broker, including
validation logic against allowedCustomMetricLabelKeys.
+* Read the topic's existing properties.
-Broker Policy Handling Logic:
+* Filter keys against the configured allow-list (merge of broker-level and
namespace-level).
-Extend the topic-level policy framework to handle customMetricLabels.
-
-Ensure changes are published to and consumed from the `__change_events` system
topic.
-
-Update broker policy caches accordingly.
-
-Metrics Generation Modification: Update the Prometheus metrics servlet (or
equivalent OTel exporter logic in the future) to retrieve custom metric labels
from the policy cache and add them to the outgoing topic metrics.
+* Append them to the outgoing Prometheus metric lines.
# Detailed Design
## Design & Implementation Details
-Topic Policy Storage: Custom metric labels will be stored as a Map<String,
String> within the topic's policy data structure.
-
-Policy Propagation and Caching: Brokers will use the existing system topic
mechanism (__change_events) for propagating custom metric label policy changes
and updating their in-memory policy caches.
+### Topic Property Storage:
+We utilize the existing Map<String, String> properties in topic metadata. No
new storage is needed.
### Metrics Generation Logic:
-The Prometheus metrics generation component in the broker (e.g.,
PrometheusMetricsServlet) will be modified.
-
-If `exposeCustomTopicMetricLabelsEnabled` is true, for each topic, it will
retrieve the customMetricLabels map from its policy cache.
-Each key-value pair from this map will be added as an additional label to all
Prometheus metrics emitted for that topic. The standard labels (cluster,
namespace, topic) will remain.
-
-Validation: Brokers will enforce maxCustomMetricLabelValueLength during the
set-custom-metric-labels operation. Keys will be validated against
allowedCustomMetricLabelKeys.
+The Prometheus metrics generation component will be modified to:
+1. Merge broker-level and namespace-level `allowedTopicPropertiesForMetrics`
(union of both)
+2. For each topic, check if its properties match any allowed key
+3. If matched, add the property as a Prometheus label
## Public-facing Changes
@@ -98,53 +73,48 @@ Description: Enables or disables the custom topic metric
labels feature.
Default: false
-allowedCustomMetricLabelKeys=<key1>,<key2>,...
-
-Description: A comma-separated list of strings defining the custom metric
label keys that administrators allow to be set on topics. Example:
sla_tier,data_sensitivity,cost_center,app_owner.
-
-Default: Empty string (if the feature is enabled but no keys are defined, no
custom metric labels can be set).
-
-maxCustomMetricLabelValueLength=<integer>
+allowedTopicPropertiesForMetrics=<key1>,<key2>,...
-Description: The maximum character length for a custom metric label value.
+Description: A comma-separated list of Topic Property keys that are allowed to
be exposed as metrics. Only keys explicitly listed here will be exposed.
-Default: 128
+Default: Empty string (if the feature is enabled but no keys are defined, no
custom metric labels can be exposed).
### Public API
-New pulsar-admin topics subcommands and corresponding REST API endpoints will
be introduced:
+#### Namespace-Level Admin API
-#### Set Custom Metric Labels:
+Namespace-level configuration allows per-namespace control of which topic
properties can be exposed as metrics. The final allowed list is the union of
broker-level and namespace-level settings.
-CLI: pulsar-admin topics set-custom-metric-labels <topic-name> --labels
"key1=value1,key2=value2"
+**REST API:**
-REST API: POST
/admin/v2/topics/{tenant}/{namespace}/{topic}/custom-metric-labels with a JSON
payload {"labels": {"key1":"value1", "key2":"value2"}}
+- `POST
/admin/v2/namespaces/{tenant}/{namespace}/allowedTopicPropertiesForMetrics` -
Set allowed properties keys
+- `GET
/admin/v2/namespaces/{tenant}/{namespace}/allowedTopicPropertiesForMetrics` -
Get allowed properties keys
+- `DELETE
/admin/v2/namespaces/{tenant}/{namespace}/allowedTopicPropertiesForMetrics` -
Remove allowed properties keys
-Action: Sets or updates custom metric labels for the specified topic.
+**CLI Examples:**
-The broker (or admin client before sending) will validate that all provided
keys (e.g., key1, key2) are present in the allowedCustomMetricLabelKeys list
defined in broker.conf.
+```bash
+# Set allowed properties keys
+pulsar-admin namespaces set-allowed-topic-properties-for-metrics \
+ --keys sla_tier,owner,environment \
+ my-tenant/my-namespace
-User cannot rewrite a label which already defined from Pulsar.
+# Get allowed properties keys
+pulsar-admin namespaces get-allowed-topic-properties-for-metrics
my-tenant/my-namespace
-Invalid keys will result in an error. This operation will update the topic's
policy and publish a change event to the system topic (__change_events) for
that namespace.
+# Remove allowed properties keys
+pulsar-admin namespaces remove-allowed-topic-properties-for-metrics
my-tenant/my-namespace
+```
-#### Get Custom Metric Labels:
+**Java Admin API:**
-CLI: pulsar-admin topics get-custom-metric-labels <topic-name>
-
-REST API: GET
/admin/v2/topics/{tenant}/{namespace}/{topic}/custom-metric-labels
-
-Action: Retrieves the currently set custom metric labels for the topic.
-
-#### Remove Custom Metric Labels:
-
-CLI:
-pulsar-admin topics remove-custom-metric-labels <topic-name> --labels
"key1,key2" (to remove specific labels)
-pulsar-admin topics remove-custom-metric-labels <topic-name> --all (to remove
all custom metric labels from the topic)
-
-REST API: DELETE
/admin/v2/topics/{tenant}/{namespace}/{topic}/custom-metric-labels with a query
params keys=k1&keys=k2 or all=true.
-
-Action: Removes the specified custom metric labels or all custom metric labels
from the topic. This also updates the topic policy.
+```java
+public interface Namespaces {
+ void setAllowedTopicPropertiesForMetrics(String namespace, Set<String>
allowedKeys) throws PulsarAdminException;
+ Set<String> getAllowedTopicPropertiesForMetrics(String namespace) throws
PulsarAdminException;
+ void removeAllowedTopicPropertiesForMetrics(String namespace) throws
PulsarAdminException;
+}
+```
# Backward & Forward Compatibility
@@ -154,15 +124,15 @@ Disabled by Default: The feature will be disabled by
default (exposeCustomTopicM
Existing Pulsar deployments will see no change in behavior or metric format.
-No Impact if Unused: If the feature is enabled but
allowedCustomMetricLabelKeys is not configured or no labels are set on topics,
metrics will remain unchanged.
+No Impact if Unused: If the feature is enabled but
allowedTopicPropertiesForMetrics is not configured or no labels are set on
topics, metrics will remain unchanged.
-Existing APIs: Existing pulsar-admin commands and REST APIs are unaffected.
The new commands and endpoints are additive.
+Existing APIs: Existing pulsar-admin commands and REST APIs are unaffected.
## Forward Compatibility
Prometheus Systems: If a Pulsar broker with this feature enabled sends metrics
with custom metric labels to an older Prometheus server or a monitoring system
not expecting these additional labels, those systems will typically ignore the
extra labels without issue.
-Future Enhancements: Future Pulsar versions could extend this feature, for
example, by allowing more dynamic management of allowedCustomMetricLabelKeys if
deemed safe and necessary.
+Future Enhancements: Future Pulsar versions could extend this feature, for
example, by allowing more dynamic management of
allowedTopicPropertiesForMetrics if deemed safe and necessary.
OpenTelemetry Alignment: The key-value structure of custom metric labels
aligns well with OpenTelemetry attributes, ensuring that this feature remains
relevant and compatible with Pulsar's evolving metrics infrastructure.
@@ -170,43 +140,27 @@ OpenTelemetry Alignment: The key-value structure of
custom metric labels aligns
A comprehensive testing strategy will be required:
-Unit Tests: For new logic in pulsar-admin, REST API handlers, policy
management, and metrics generation.
+Test the filtering logic against allowedTopicPropertiesForMetrics.
Integration Tests:
Verify correct setting, getting, and removing of custom metric labels via
admin tools and REST APIs.
-Test validation logic for allowed keys, max labels per topic, and value length.
-
-Ensure correct propagation of custom metric label policy changes via system
topics and updates to broker policy caches.
+Test validation logic for allowed keys.
-Verify that Prometheus metrics output accurately includes the custom metric
labels for relevant topics and does not include them when the feature is
disabled or labels are not set.
+End-to-End Flow: Set a property via pulsar-admin topics update-properties,
scrape the metrics endpoint, and verify the label appears.
-End-to-End Tests: Simulate a Pulsar cluster environment, set custom metric
labels on topics, and scrape metrics using a Prometheus instance to confirm
labels appear correctly and can be queried.
+Dynamic Updates: Verify that updating a property value changes the metric
label value in subsequent scrapes.
-Performance Tests: Assess any potential performance impact on brokers,
particularly concerning policy updates and metrics generation, especially in
environments with a large number of topics.
+Removal: Verify that removing a property removes the label.
# Documentation Plan
-The official Apache Pulsar documentation will be updated to include:
-
-Concepts Section: Explanation of the custom topic metric labels feature, its
purpose, and how it helps with monitoring and alerting.
-
-Administrator Guide:
+Documentation updates will include:
-Instructions on how to enable and configure the feature in broker.conf (e.g.,
exposeCustomTopicMetricLabelsEnabled, allowedCustomMetricLabelKeys, and other
limits).
-
-Best practices for defining allowedCustomMetricLabelKeys with cardinality
management in mind.
-
-User Guide / pulsar-admin Reference:
-
-Detailed syntax and examples for the new pulsar-admin topics
set/get/remove-custom-metric-labels commands.
-
-Guidance on choosing appropriate values for predefined keys to manage
cardinality.
-
-REST API Reference: Documentation for the new REST API endpoints.
-
-Monitoring Section: Notes on how these custom metric labels appear in
Prometheus and how they can be used in PromQL queries, along with reminders
about cardinality considerations for the Prometheus system.
+- **Configuration Guide**: How to configure broker-level and namespace-level
`allowedTopicPropertiesForMetrics`
+- **Admin API Guide**: How to use pulsar-admin commands to manage
namespace-level settings
+- **Monitoring Guide**: How custom labels appear in Prometheus and best
practices for cardinality management
# Alternatives
@@ -224,6 +178,12 @@ Description: Keeping Pulsar metrics unchanged and using
Prometheus's relabel_con
Reason for Rejection: This shifts the implementation complexity and
maintenance burden to the Prometheus configuration and external systems. It
introduces risks of stale metadata and potential performance overhead on
Prometheus. Crucially, it is not a Pulsar-native solution, which is the aim of
this proposal.
+C. Store label in topic policy
+
+Description: Instead of using topic properties, we can store the custom metric
labels in topic policies.
+
+Reason for Rejection: Storing labels in topic policies adds complexity to the
topic management and requires additional code to handle the policy updates.
+
# Links
* Mailing List discussion thread:
https://lists.apache.org/thread/66l8cdhx5f7sv05mqfnlwc7s570frtzq
