This is an automated email from the ASF dual-hosted git repository. technoboy pushed a commit to branch release-3.2.0 in repository https://gitbox.apache.org/repos/asf/pulsar-site.git
commit 895b004e0786876bfcf6a2568c6e010ad16013e2 Author: Asaf Mesika <[email protected]> AuthorDate: Wed Jan 24 13:34:15 2024 +0200 Added docs for PIP-323 (#773) --- docs/administration-stats.md | 58 +++--- docs/cookbooks-retention-expiry.md | 21 ++- docs/reference-metrics.md | 353 +++++++++++++++++++------------------ docs/reference-terminology.md | 11 ++ 4 files changed, 234 insertions(+), 209 deletions(-) diff --git a/docs/administration-stats.md b/docs/administration-stats.md index 073f3be00f95..27aa699ef1ba 100644 --- a/docs/administration-stats.md +++ b/docs/administration-stats.md @@ -15,32 +15,36 @@ All stats below are **reset** to 0 upon broker restart or topic unloading, **exc ::: -| Stat | Description | -|--------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------| -| msgRateIn | The sum of all local and replication publishers' publish rates (message per second). | -| msgThroughputIn | The sum of all local and replication publishers' publish rates (byte per second). | -| msgRateOut | The sum of all local and replication consumers' dispatch rates (message per second). | -| msgThroughputOut | The sum of all local and replication consumers' dispatch rates (byte per second). | -| averageMsgSize | The average size (bytes) of messages published within the last interval. | -| storageSize* | The sum of the ledgers' storage size **in BookKeeper** for a topic (in bytes). <br/><br/>**Note**: the `total storage size of a topic` = `storageSize` + `offloadedStorageSize`. | -| offloadedStorageSize* | The sum of the storage size **in tiered storage** for a topic (in bytes).<br/><br/>**Note**: the `total storage size of a topic` = `storageSize` + `offloadedStorageSize`. -| earliestMsgPublishTimeInBacklogs* | The publish time of the earliest message in the backlog (in milliseconds). | -| bytesInCounter | The total bytes published to the topic. | -| msgInCounter | The total messages published to the topic. | -| bytesOutCounter | The total bytes delivered to consumers. | -| msgOutCounter | The total messages delivered to consumers. | -| msgChunkPublished | The topics that have chunked messages published on it. | -| backlogSize* | The estimated total unconsumed or backlog size (in bytes). | -| waitingPublishers | The number of publishers waiting in a queue in exclusive access mode. | -| deduplicationStatus | The status of message deduplication for the topic. | -| topicEpoch | The topic epoch or empty if not set. | -| filteredEntriesCount | The count of skipped entries for the topic. | -| nonContiguousDeletedMessagesRanges | The number of non-contiguous deleted messages ranges. | -| nonContiguousDeletedMessagesRangesSerializedSize | The serialized size of non-contiguous deleted messages ranges. | -| ownerBroker | The broker that owns this topic. | -| [publishers](#producer-stats) | The list of all local publishers on the topic, ranging from zero to thousands. | -| [subscriptions](#subscription-stats) | The list of all local subscriptions to the topic. | -| [replication](#replication-stats) | This section gives the stats for cross-colo replication of this topic. | +| Stat | Description | +|--------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| msgRateIn | The sum of all local and replication publishers' publish rates (message per second). | +| msgThroughputIn | The sum of all local and replication publishers' publish rates (byte per second). | +| msgRateOut | The sum of all local and replication consumers' dispatch rates (message per second). | +| msgThroughputOut | The sum of all local and replication consumers' dispatch rates (byte per second). | +| averageMsgSize | The average size (bytes) of messages published within the last interval. | +| storageSize* | The sum of the ledgers' storage size **in BookKeeper** for a topic (in bytes). <br/><br/>**Note**: the `total storage size of a topic` = `storageSize` + `offloadedStorageSize`. | +| offloadedStorageSize* | The sum of the storage size **in tiered storage** for a topic (in bytes).<br/><br/>**Note**: the `total storage size of a topic` = `storageSize` + `offloadedStorageSize`. | +| earliestMsgPublishTimeInBacklogs* | The publish time of the earliest message in the backlog (in milliseconds). | +| bytesInCounter | The total bytes published to the topic. | +| msgInCounter | The total messages published to the topic. | +| bytesOutCounter | The total bytes delivered to consumers. | +| msgOutCounter | The total messages delivered to consumers. | +| msgChunkPublished | The topics that have chunked messages published on it. | +| backlogSize* | The estimated total unconsumed or backlog size (in bytes). | +| oldestBacklogMessageAgeSeconds | The age of the oldest unacknowledged (i.e. backlog) message, measured by the time elapsed from its published time, in seconds. This value is recorded every backlog quota check interval, hence it represents the value seen in the last check. | +| oldestBacklogMessageSubscriptionName | The name of the subscription containing the oldest unacknowledged message. This value is recorded every backlog quota check interval, hence it represents the value seen in the last check. | +| backlogQuotaLimitSize | the size in bytes of the [topic backlog quota](./cookbooks-retention-expiry.md#backlog-quotas). | +| backlogQuotaLimitTime | the [topic backlog age (time) quota](./cookbooks-retention-expiry.md#backlog-quotas), in seconds. | +| waitingPublishers | The number of publishers waiting in a queue in exclusive access mode. | +| deduplicationStatus | The status of message deduplication for the topic. | +| topicEpoch | The topic epoch or empty if not set. | +| filteredEntriesCount | The count of skipped entries for the topic. | +| nonContiguousDeletedMessagesRanges | The number of non-contiguous deleted messages ranges. | +| nonContiguousDeletedMessagesRangesSerializedSize | The serialized size of non-contiguous deleted messages ranges. | +| ownerBroker | The broker that owns this topic. | +| [publishers](#producer-stats) | The list of all local publishers on the topic, ranging from zero to thousands. | +| [subscriptions](#subscription-stats) | The list of all local subscriptions to the topic. | +| [replication](#replication-stats) | This section gives the stats for cross-colo replication of this topic. | ### Producer stats @@ -175,4 +179,4 @@ The following table outlines the internal stats inside a topic. For more details | entries | The total number of entries that belong to this ledger. | | size | The size of messages written to this ledger (in bytes). | | offloaded | Whether this ledger is offloaded. The value is `false` for the compacted topic ledger. | -| metadata | The ledger metadata. | \ No newline at end of file +| metadata | The ledger metadata. | diff --git a/docs/cookbooks-retention-expiry.md b/docs/cookbooks-retention-expiry.md index ad7408ff4fe7..719d54a94b8c 100644 --- a/docs/cookbooks-retention-expiry.md +++ b/docs/cookbooks-retention-expiry.md @@ -61,7 +61,7 @@ When a retention limit on a topic is exceeded, the oldest message is marked for You can set message retention at instance level with the following two parameters: `defaultRetentionTimeInMinutes` and `defaultRetentionSizeInMB`. By default, both parameters are set to `0`, which means disabling message retention. -For more information on the two parameters, refer to the [`broker.conf`](reference-configuration.md#broker) configuration file. +For more information on the two parameters, refer to the [`broker.conf`](reference-configuration.md) configuration file. ### Set retention policy @@ -198,12 +198,11 @@ Setting a backlog quota involves setting: The following retention policies are available: -Policy | Action -:------|:------ -`producer_request_hold` | The producer holds the message and retris until client configuration `sendTimeoutMs` is exceeded. -`producer_exception` | The producer throws an exception when trying to send a message -`consumer_backlog_eviction` | The broker will begin discarding backlog messages - +| Policy | Action | +|:----------------------------|:--------------------------------------------------------------------------------------------------| +| `producer_request_hold` | The producer holds the message and retris until client configuration `sendTimeoutMs` is exceeded. | +| `producer_exception` | The producer throws an exception when trying to send a message | +| `consumer_backlog_eviction` | The broker will begin discarding backlog messages | > #### Beware the distinction between retention policy types > As you may have noticed, there are two definitions of the term "retention > policy" in Pulsar, one that applies to persistent storage of messages not in > backlogs, and one that applies to messages within backlogs. @@ -213,7 +212,7 @@ Backlog quotas are handled at the namespace level. They can be managed via: ### Set size/time thresholds and backlog retention policies -You can set a size and/or time threshold and backlog retention policy for all of the topics in a [namespace](reference-terminology.md#namespace) by specifying the namespace, a size limit and/or a time limit in second, and a policy by name. +You can set a size and/or time threshold and backlog retention policy for all the topics in a [namespace](reference-terminology.md#namespace) by specifying the namespace, a size limit and/or a time limit in second, and a policy by name. ````mdx-code-block <Tabs groupId="api-choice" @@ -298,6 +297,12 @@ Map<BacklogQuota.BacklogQuotaType,BacklogQuota> quotas = </Tabs> ```` +### Monitoring backlog quota usage + +The backlog quota limit and usage are exposed both through [metrics](./reference-metrics.md#topic-metrics), +and [topic stats](./administration-stats.md#topic-stats). This allows to define, for example to define alerts +to know ahead of time if a backlog quota is about to reach its limit. + ### Remove backlog quotas diff --git a/docs/reference-metrics.md b/docs/reference-metrics.md index 54af9d70ace6..d30a60f4f428 100644 --- a/docs/reference-metrics.md +++ b/docs/reference-metrics.md @@ -8,7 +8,7 @@ sidebar_label: "Metrics" Pulsar exposes the following metrics in Prometheus format. You can monitor your clusters with those metrics. -- [](#) +- [](./reference-metrics.md) - [Pulsar Functions](#pulsar-functions) - [Connectors](#connectors) - [Proxy](#proxy) @@ -120,46 +120,48 @@ All the metrics exposed by a broker are labeled with `cluster=${pulsar_cluster}` All the broker metrics are labeled with the following labels: - cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the `broker.conf` file. -| Name | Type | Description | -|---|---|---| -| pulsar_ml_cache_evictions | Gauge | The number of cache evictions during the last minute. | -| pulsar_ml_cache_inserted_entries_total | Counter | The number of entries inserted into the entry cache. | -| pulsar_ml_cache_evicted_entries_total | Counter | The number of entries evicted from the entry cache. | -| pulsar_ml_cache_entries | Gauge | The number of entries in the entry cache. | -| pulsar_ml_cache_hits_rate | Gauge | The number of cache hits per second on the broker side. | -| pulsar_ml_cache_hits_throughput | Gauge | The amount of data (byte per second) retrieved from the cache on the broker side. | -| pulsar_ml_cache_misses_rate | Gauge | The number of cache missed per second on the broker side. | -| pulsar_ml_cache_misses_throughput | Gauge | The amount of data (byte per second) that cannot be retrieved from the cache on the broker side. | -| pulsar_ml_cache_pool_active_allocations | Gauge | The number of currently active allocations in direct arena. | -| pulsar_ml_cache_pool_active_allocations_huge | Gauge | The number of currently active huge allocation in direct arena. | -| pulsar_ml_cache_pool_active_allocations_normal | Gauge | The number of currently active normal allocations in direct arena. | -| pulsar_ml_cache_pool_active_allocations_small | Gauge | The number of currently active small allocations in direct arena. | -| pulsar_ml_cache_pool_allocated | Gauge | The total allocated memory of chunk lists in direct arena. | -| pulsar_ml_cache_pool_used | Gauge | The total used memory of chunk lists in direct arena. | -| pulsar_ml_cache_used_size | Gauge | The size used to store the payloads of entries (in bytes). | -| pulsar_ml_count | Gauge | The number of currently opened managed ledgers. | -| ~~topic_load_times (deprecated)~~ | ~~Summary~~ | ~~The topic load latency calculated in milliseconds.~~ | -| pulsar_topic_load_times | Summary | The topic load latency calculated in milliseconds. | -| pulsar_active_connections| Gauge | The number of active connections. | -| pulsar_connection_created_total_count | Gauge | The total number of connections. | -| pulsar_connection_create_success_count | Gauge | The number of successfully created connections. | -| pulsar_connection_create_fail_count | Gauge | The number of failed connections. | -| pulsar_connection_closed_total_count | Gauge | The total number of closed connections. | -| pulsar_broker_throttled_connections | Gauge | The number of throttled connections. | -| pulsar_broker_throttled_connections_global_limit | Gauge | The number of throttled connections due to per-connection limit. | -| pulsar_broker_topics_count | Gauge | The number of Pulsar topics in this broker. | -| pulsar_broker_subscriptions_count | Gauge | The number of Pulsar subscriptions in this broker. | -| pulsar_broker_producers_count | Gauge | The number of active producers connected to this broker. | -| pulsar_broker_consumers_count | Gauge | The number of active consumers connected to this broker. | -| pulsar_broker_rate_in | Gauge | The total message rate coming into this broker (message per second). | -| pulsar_broker_rate_out | Gauge | The total message rate going out from this broker (message per second). | -| pulsar_broker_throughput_in | Gauge | The total throughput coming into this broker (byte per second). | -| pulsar_broker_throughput_out | Gauge | The total throughput going out from this broker (byte per second). | -| pulsar_broker_storage_size | Gauge | The total storage size of all topics in this broker (in bytes). | -| pulsar_broker_storage_logical_size | Gauge | The storage size of all topics in this broker without replicas (in bytes). | -| pulsar_broker_storage_write_rate | Gauge | The total message batches (entries) written to the storage for this broker (message batch per second). | -| pulsar_broker_storage_read_rate | Gauge | The total message batches (entries) read from the storage for this broker (message batch per second). | -| pulsar_broker_msg_backlog | Gauge | The total number of message backlogs in this broker (entries). | +| Name | Type | Description | +|--------------------------------------------------------------|-------------|-----------------------------------------------------------------------------------------------------------------------------| +| pulsar_ml_cache_evictions | Gauge | The number of cache evictions during the last minute. | +| pulsar_ml_cache_inserted_entries_total | Counter | The number of entries inserted into the entry cache. | +| pulsar_ml_cache_evicted_entries_total | Counter | The number of entries evicted from the entry cache. | +| pulsar_ml_cache_entries | Gauge | The number of entries in the entry cache. | +| pulsar_ml_cache_hits_rate | Gauge | The number of cache hits per second on the broker side. | +| pulsar_ml_cache_hits_throughput | Gauge | The amount of data (byte per second) retrieved from the cache on the broker side. | +| pulsar_ml_cache_misses_rate | Gauge | The number of cache missed per second on the broker side. | +| pulsar_ml_cache_misses_throughput | Gauge | The amount of data (byte per second) that cannot be retrieved from the cache on the broker side. | +| pulsar_ml_cache_pool_active_allocations | Gauge | The number of currently active allocations in direct arena. | +| pulsar_ml_cache_pool_active_allocations_huge | Gauge | The number of currently active huge allocation in direct arena. | +| pulsar_ml_cache_pool_active_allocations_normal | Gauge | The number of currently active normal allocations in direct arena. | +| pulsar_ml_cache_pool_active_allocations_small | Gauge | The number of currently active small allocations in direct arena. | +| pulsar_ml_cache_pool_allocated | Gauge | The total allocated memory of chunk lists in direct arena. | +| pulsar_ml_cache_pool_used | Gauge | The total used memory of chunk lists in direct arena. | +| pulsar_ml_cache_used_size | Gauge | The size used to store the payloads of entries (in bytes). | +| pulsar_ml_count | Gauge | The number of currently opened managed ledgers. | +| ~~topic_load_times (deprecated)~~ | ~~Summary~~ | ~~The topic load latency calculated in milliseconds.~~ | +| pulsar_topic_load_times | Summary | The topic load latency calculated in milliseconds. | +| pulsar_active_connections | Gauge | The number of active connections. | +| pulsar_connection_created_total_count | Gauge | The total number of connections. | +| pulsar_connection_create_success_count | Gauge | The number of successfully created connections. | +| pulsar_connection_create_fail_count | Gauge | The number of failed connections. | +| pulsar_connection_closed_total_count | Gauge | The total number of closed connections. | +| pulsar_broker_throttled_connections | Gauge | The number of throttled connections. | +| pulsar_broker_throttled_connections_global_limit | Gauge | The number of throttled connections due to per-connection limit. | +| pulsar_broker_topics_count | Gauge | The number of Pulsar topics in this broker. | +| pulsar_broker_subscriptions_count | Gauge | The number of Pulsar subscriptions in this broker. | +| pulsar_broker_producers_count | Gauge | The number of active producers connected to this broker. | +| pulsar_broker_consumers_count | Gauge | The number of active consumers connected to this broker. | +| pulsar_broker_rate_in | Gauge | The total message rate coming into this broker (message per second). | +| pulsar_broker_rate_out | Gauge | The total message rate going out from this broker (message per second). | +| pulsar_broker_throughput_in | Gauge | The total throughput coming into this broker (byte per second). | +| pulsar_broker_throughput_out | Gauge | The total throughput going out from this broker (byte per second). | +| pulsar_broker_storage_size | Gauge | The total storage size of all topics in this broker (in bytes). | +| pulsar_broker_storage_logical_size | Gauge | The storage size of all topics in this broker without replicas (in bytes). | +| pulsar_broker_storage_write_rate | Gauge | The total message batches (entries) written to the storage for this broker (message batch per second). | +| pulsar_broker_storage_read_rate | Gauge | The total message batches (entries) read from the storage for this broker (message batch per second). | +| pulsar_broker_msg_backlog | Gauge | The total number of message backlogs in this broker (entries). | +| pulsar_storage_backlog_quota_check_duration_seconds | Histogram | The duration of the backlog quota check process (in seconds) | +| pulsar_broker_storage_backlog_quota_exceeded_evictions_total | Counter | The number of times a backlog was evicted since it has exceeded its quota. Includes the label `quota_type = (time \| size)` | ### BookKeeper client metrics @@ -188,32 +190,33 @@ All the namespace metrics are labeled with the following labels: - *cluster*: `cluster=${pulsar_cluster}`. `${pulsar_cluster}` is the cluster name that you configured in `broker.conf`. - *namespace*: `namespace=${pulsar_namespace}`. `${pulsar_namespace}` is the namespace name. -| Name | Type | Description | -|---|---|---| -| pulsar_topics_count | Gauge | The number of Pulsar topics of the namespace owned by this broker. | -| pulsar_subscriptions_count | Gauge | The number of Pulsar subscriptions of the namespace served by this broker. | -| pulsar_producers_count | Gauge | The number of active producers of the namespace connected to this broker. | -| pulsar_consumers_count | Gauge | The number of active consumers of the namespace connected to this broker. | -| pulsar_rate_in | Gauge | The total message rate of the namespace coming into this broker (message per second). | -| pulsar_rate_out | Gauge | The total message rate of the namespace going out from this broker (message per second). | -| pulsar_throughput_in | Gauge | The total throughput of the namespace coming into this broker (byte per second). | -| pulsar_throughput_out | Gauge | The total throughput of the namespace going out from this broker (byte per second). | -| pulsar_consumer_msg_ack_rate | Gauge | The total message acknowledgment rate of the namespace owned by this broker (message per second). | -| pulsar_storage_size | Gauge | The total storage size of the topics in this namespace owned by this broker (bytes). | -| pulsar_storage_logical_size | Gauge | The storage size of topics in the namespace owned by the broker without replicas (in bytes). | -| pulsar_storage_backlog_size | Gauge | The total backlog size of the topics of this namespace owned by this broker (in bytes). | -| pulsar_storage_offloaded_size | Gauge | The total amount of the data in this namespace offloaded to the tiered storage (bytes). | -| pulsar_storage_write_rate | Gauge | The total message batches (entries) written to the storage for this namespace (message batch per second). | -| pulsar_storage_read_rate | Gauge | The total message batches (entries) read from the storage for this namespace (message batch per second). | -| pulsar_subscription_delayed | Gauge | The total message batches (entries) are delayed for dispatching. | -| pulsar_storage_write_latency_le_* | Histogram | The entry rate of a namespace that the storage write latency is smaller with a given threshold.<br /> Available thresholds: <br /><ul><li>pulsar_storage_write_latency_le_0_5: <= 0.5ms </li><li>pulsar_storage_write_latency_le_1: <= 1ms</li><li>pulsar_storage_write_latency_le_5: <= 5ms</li><li>pulsar_storage_write_latency_le_10: <= 10ms</li><li>pulsar_storage_write_latency_le_20: <= 20ms</li><li>pulsar_storage_write_latency_le_50: <= 50ms</ [...] -| pulsar_entry_size_le_* | Histogram | The entry rate of a namespace that the entry size is smaller with a given threshold.<br /> Available thresholds: <br /><ul><li>pulsar_entry_size_le_128: <= 128 bytes </li><li>pulsar_entry_size_le_512: <= 512 bytes</li><li>pulsar_entry_size_le_1_kb: <= 1 KB</li><li>pulsar_entry_size_le_2_kb: <= 2 KB</li><li>pulsar_entry_size_le_4_kb: <= 4 KB</li><li>pulsar_entry_size_le_16_kb: <= 16 KB</li><li>pulsar_entry_size_le_100_kb: <= 100 KB</li><li>pulsar_ent [...] -| pulsar_delayed_message_index_size_bytes | Gauge | The total memory size allocated by `DelayedDeliveryTracker` of the namespace owned by this broker (in bytes). | -| pulsar_delayed_message_index_bucket_total | Gauge | The number of delayed message index buckets (immutable buckets + LastMutableBucket ) | -| pulsar_delayed_message_index_loaded | Gauge | The total number of delayed message indexes for in the memory. | -| pulsar_delayed_message_index_bucket_snapshot_size_bytes | Gauge | The total size of delayed message index bucket snapshot (in bytes). | -| pulsar_delayed_message_index_bucket_op_count | Counter | The total number of operation delayed message index bucket snapshots. The `state` label can be `succeed`,`failed`, and`all` (`all` means the total number of all states) and the `type` label can be `create`,`load`,`delete`, and `merge`. | -| pulsar_delayed_message_index_bucket_op_latency_ms | Histogram | The latency of delayed message index bucket snapshot operation with a given quantile (threshold). The `type` label can be `create`,`load`,`delete`, and `merge`<br/>The label `quantile` can be:<ul><li>quantile="50" is operation latency between (0ms, 50ms]</li><li>quantile="100" is operation latency between (50ms, 100ms]</li><li>quantile="500" is operation latency between (100ms, 500ms]</li><li>quantile="1000" is operation l [...] +| Name | Type | Description [...] +|---------------------------------------------------------|-----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...] +| pulsar_topics_count | Gauge | The number of Pulsar topics of the namespace owned by this broker. [...] +| pulsar_subscriptions_count | Gauge | The number of Pulsar subscriptions of the namespace served by this broker. [...] +| pulsar_producers_count | Gauge | The number of active producers of the namespace connected to this broker. [...] +| pulsar_consumers_count | Gauge | The number of active consumers of the namespace connected to this broker. [...] +| pulsar_rate_in | Gauge | The total message rate of the namespace coming into this broker (message per second). [...] +| pulsar_rate_out | Gauge | The total message rate of the namespace going out from this broker (message per second). [...] +| pulsar_throughput_in | Gauge | The total throughput of the namespace coming into this broker (byte per second). [...] +| pulsar_throughput_out | Gauge | The total throughput of the namespace going out from this broker (byte per second). [...] +| pulsar_consumer_msg_ack_rate | Gauge | The total message acknowledgment rate of the namespace owned by this broker (message per second). [...] +| pulsar_storage_size | Gauge | The total storage size of the topics in this namespace owned by this broker (bytes). [...] +| pulsar_storage_logical_size | Gauge | The storage size of topics in the namespace owned by the broker without replicas (in bytes). [...] +| pulsar_storage_backlog_size | Gauge | The total backlog size of the topics of this namespace owned by this broker (in bytes). [...] +| pulsar_storage_backlog_quota_exceeded_evictions_total | Counter | The number of times a backlog was evicted since it has exceeded its quota (When [backlog quota's retention policy](./cookbooks-retention-expiry.md#backlog-quotas) was set to `consumer_backlog_eviction`). Also includes label `quota_type = (time \| size)` [...] +| pulsar_storage_offloaded_size | Gauge | The total amount of the data in this namespace offloaded to the tiered storage (bytes). [...] +| pulsar_storage_write_rate | Gauge | The total message batches (entries) written to the storage for this namespace (message batch per second). [...] +| pulsar_storage_read_rate | Gauge | The total message batches (entries) read from the storage for this namespace (message batch per second). [...] +| pulsar_subscription_delayed | Gauge | The total message batches (entries) are delayed for dispatching. [...] +| pulsar_storage_write_latency_le_* | Histogram | The entry rate of a namespace that the storage write latency is smaller with a given threshold.<br /> Available thresholds: <br /><ul><li>pulsar_storage_write_latency_le_0_5: <= 0.5ms </li><li>pulsar_storage_write_latency_le_1: <= 1ms</li><li>pulsar_storage_write_latency_le_5: <= 5ms</li><li>pulsar_storage_write_latency_le_10: <= 10ms</li><li>pulsar_storage_write_latency_le_20: <= 20ms</li><li>pulsar_storage_write_la [...] +| pulsar_entry_size_le_* | Histogram | The entry rate of a namespace that the entry size is smaller with a given threshold.<br /> Available thresholds: <br /><ul><li>pulsar_entry_size_le_128: <= 128 bytes </li><li>pulsar_entry_size_le_512: <= 512 bytes</li><li>pulsar_entry_size_le_1_kb: <= 1 KB</li><li>pulsar_entry_size_le_2_kb: <= 2 KB</li><li>pulsar_entry_size_le_4_kb: <= 4 KB</li><li>pulsar_entry_size_le_16_kb: <= 16 KB</li><li>pulsar_entry_size_le_100 [...] +| pulsar_delayed_message_index_size_bytes | Gauge | The total memory size allocated by `DelayedDeliveryTracker` of the namespace owned by this broker (in bytes). [...] +| pulsar_delayed_message_index_bucket_total | Gauge | The number of delayed message index buckets (immutable buckets + LastMutableBucket ) [...] +| pulsar_delayed_message_index_loaded | Gauge | The total number of delayed message indexes for in the memory. [...] +| pulsar_delayed_message_index_bucket_snapshot_size_bytes | Gauge | The total size of delayed message index bucket snapshot (in bytes). [...] +| pulsar_delayed_message_index_bucket_op_count | Counter | The total number of operation delayed message index bucket snapshots. The `state` label can be `succeed`,`failed`, and`all` (`all` means the total number of all states) and the `type` label can be `create`,`load`,`delete`, and `merge`. [...] +| pulsar_delayed_message_index_bucket_op_latency_ms | Histogram | The latency of delayed message index bucket snapshot operation with a given quantile (threshold). The `type` label can be `create`,`load`,`delete`, and `merge`<br/>The label `quantile` can be:<ul><li>quantile="50" is operation latency between (0ms, 50ms]</li><li>quantile="100" is operation latency between (50ms, 100ms]</li><li>quantile="500" is operation latency between (100ms, 500ms]</li><li>quantile="1000" is opera [...] ### Topic metrics @@ -225,49 +228,51 @@ All the topic metrics are labeled with the following labels: - *namespace*: `namespace=${pulsar_namespace}`. `${pulsar_namespace}` is the namespace name. - *topic*: `topic=${pulsar_topic}`. `${pulsar_topic}` is the topic name. -| Name | Type | Description | -|---|---|---| -| pulsar_subscriptions_count | Gauge | The number of Pulsar subscriptions of the topic served by this broker. | -| pulsar_producers_count | Gauge | The number of active producers of the topic connected to this broker. | -| pulsar_consumers_count | Gauge | The number of active consumers of the topic connected to this broker. | -| pulsar_rate_in | Gauge | The total message rate of the topic coming into this broker (message per second). | -| pulsar_rate_out | Gauge | The total message rate of the topic going out from this broker (message per second). | -| pulsar_publish_rate_limit_times | Gauge | The number of times the publish rate limit is triggered. | -| pulsar_throughput_in | Gauge | The total throughput of the topic coming into this broker (byte per second). | -| pulsar_throughput_out | Gauge | The total throughput of the topic going out from this broker (byte per second). | -| pulsar_consumer_msg_ack_rate | Gauge | The total message acknowledgment rate of the topic connected to this broker (message per second). | -| pulsar_storage_size | Gauge | The total storage size of the topics in this topic owned by this broker (bytes). | -| pulsar_storage_logical_size | Gauge | The storage size of topics in the namespace owned by the broker without replicas (in bytes). | -| pulsar_storage_backlog_size | Gauge | The total backlog size of the topics of this topic owned by this broker (in bytes). | -| pulsar_storage_offloaded_size | Gauge | The total amount of the data in this topic offloaded to the tiered storage (bytes). | -| pulsar_storage_backlog_quota_limit | Gauge | The total amount of the data in this topic that limit the backlog quota (bytes). | -| pulsar_storage_write_rate | Gauge | The total message batches (entries) written to the storage for this topic (message batch per second). | -| pulsar_storage_read_rate | Gauge | The total message batches (entries) read from the storage for this topic (message batch per second). | -| pulsar_subscription_delayed | Gauge | The total message batches (entries) are delayed for dispatching. | -| pulsar_storage_write_latency_le_* | Histogram | The entry rate of a topic that the storage write latency is smaller with a given threshold.<br /> Available thresholds: <br /><ul><li>pulsar_storage_write_latency_le_0_5: <= 0.5ms </li><li>pulsar_storage_write_latency_le_1: <= 1ms</li><li>pulsar_storage_write_latency_le_5: <= 5ms</li><li>pulsar_storage_write_latency_le_10: <= 10ms</li><li>pulsar_storage_write_latency_le_20: <= 20ms</li><li>pulsar_storage_write_latency_le_50: <= 50ms</li>< [...] -| pulsar_entry_size_le_* | Histogram | The entry rate of a topic that the entry size is smaller with a given threshold.<br /> Available thresholds: <br /><ul><li>pulsar_entry_size_le_128: <= 128 bytes </li><li>pulsar_entry_size_le_512: <= 512 bytes</li><li>pulsar_entry_size_le_1_kb: <= 1 KB</li><li>pulsar_entry_size_le_2_kb: <= 2 KB</li><li>pulsar_entry_size_le_4_kb: <= 4 KB</li><li>pulsar_entry_size_le_16_kb: <= 16 KB</li><li>pulsar_entry_size_le_100_kb: <= 100 KB</li><li>pulsar_entry_s [...] -| pulsar_in_bytes_total | Counter | The total number of messages in bytes received for this topic. | -| pulsar_in_messages_total | Counter | The total number of messages received for this topic. | -| pulsar_out_bytes_total | Counter | The total number of messages in bytes read from this topic. | -| pulsar_out_messages_total | Counter | The total number of messages read from this topic. | -| pulsar_compaction_removed_event_count | Gauge | The total number of removed events of the compaction. | -| pulsar_compaction_succeed_count | Gauge | The total number of successes of the compaction. | -| pulsar_compaction_failed_count | Gauge | The total number of failures of the compaction. | -| pulsar_compaction_duration_time_in_mills | Gauge | The duration time of the compaction. | -| pulsar_compaction_read_throughput | Gauge | The read throughput of the compaction. | -| pulsar_compaction_write_throughput | Gauge | The write throughput of the compaction. | -| pulsar_compaction_latency_le_* | Histogram | The compaction latency with given quantile. <br /> Available thresholds: <br /><ul><li>pulsar_compaction_latency_le_0_5: <= 0.5ms </li><li>pulsar_compaction_latency_le_1: <= 1ms</li><li>pulsar_compaction_latency_le_5: <= 5ms</li><li>pulsar_compaction_latency_le_10: <= 10ms</li><li>pulsar_compaction_latency_le_20: <= 20ms</li><li>pulsar_compaction_latency_le_50: <= 50ms</li><li>pulsar_compaction_latency_le_100: <= 100ms</li><li>pulsar_compact [...] -| pulsar_compaction_compacted_entries_count | Gauge | The total number of the compacted entries. | -| pulsar_compaction_compacted_entries_size |Gauge | The total size of the compacted entries. | -| pulsar_txn_tb_active_total | Gauge | The number of active transactions on this topic. | -| pulsar_txn_tb_aborted_total | Counter | The number of aborted transactions on the topic. | -| pulsar_txn_tb_committed_total | Counter | The number of committed transactions on the topic. | -| pulsar_delayed_message_index_size_bytes | Gauge | The total memory size allocated by `DelayedDeliveryTracker` of the topic owned by this broker (in bytes). | -| pulsar_delayed_message_index_bucket_total | Gauge | The number of delayed message index buckets (immutable buckets + LastMutableBucket ) | -| pulsar_delayed_message_index_loaded | Gauge | The total number of delayed message indexes for in the memory. | -| pulsar_delayed_message_index_bucket_snapshot_size_bytes | Gauge | The total size of delayed message index bucket snapshot (in bytes). | -| pulsar_delayed_message_index_bucket_op_count | Counter | The total number of operation delayed message index bucket snapshots. The `state` label can be `succeed`,`failed`, and`all` (`all` means the total number of all states) and the `type` label can be `create`,`load`,`delete`, and `merge`. | -| pulsar_delayed_message_index_bucket_op_latency_ms | Histogram | The latency of delayed message index bucket snapshot operation with a given quantile (threshold). The label`type` label can be `create`,`load`,`delete`, and `merge`<br/>The label `quantile` can be:<ul><li>quantile="50" is operation latency between (0ms, 50ms]</li><li>quantile="100" is operation latency between (50ms, 100ms]</li><li>quantile="500" is operation latency between (100ms, 500ms]</li><li>quantile="1000" is operat [...] +| Name | Type | Description [...] +|---------------------------------------------------------|-----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...] +| pulsar_subscriptions_count | Gauge | The number of Pulsar subscriptions of the topic served by this broker. [...] +| pulsar_producers_count | Gauge | The number of active producers of the topic connected to this broker. [...] +| pulsar_consumers_count | Gauge | The number of active consumers of the topic connected to this broker. [...] +| pulsar_rate_in | Gauge | The total message rate of the topic coming into this broker (message per second). [...] +| pulsar_rate_out | Gauge | The total message rate of the topic going out from this broker (message per second). [...] +| pulsar_publish_rate_limit_times | Gauge | The number of times the publish rate limit is triggered. [...] +| pulsar_throughput_in | Gauge | The total throughput of the topic coming into this broker (byte per second). [...] +| pulsar_throughput_out | Gauge | The total throughput of the topic going out from this broker (byte per second). [...] +| pulsar_consumer_msg_ack_rate | Gauge | The total message acknowledgment rate of the topic connected to this broker (message per second). [...] +| pulsar_storage_size | Gauge | The total storage size of the topics in this topic owned by this broker (bytes). [...] +| pulsar_storage_logical_size | Gauge | The storage size of topics in the namespace owned by the broker without replicas (in bytes). [...] +| pulsar_storage_backlog_size | Gauge | The total backlog size of the topics of this topic owned by this broker (in bytes). [...] +| pulsar_storage_offloaded_size | Gauge | The total amount of the data in this topic offloaded to the tiered storage (bytes). [...] +| pulsar_storage_backlog_quota_limit | Gauge | The total amount of the data in this topic that limit the backlog quota (bytes). [...] +| pulsar_storage_backlog_age_seconds | Gauge | The age of the oldest unacknowledged message (backlog). [...] +| pulsar_storage_backlog_quota_exceeded_evictions_total | Counter | The number of times a backlog was evicted since it has exceeded its quota. Includes label `quota_type = (time \| size)` [...] +| pulsar_storage_write_rate | Gauge | The total message batches (entries) written to the storage for this topic (message batch per second). [...] +| pulsar_storage_read_rate | Gauge | The total message batches (entries) read from the storage for this topic (message batch per second). [...] +| pulsar_subscription_delayed | Gauge | The total message batches (entries) are delayed for dispatching. [...] +| pulsar_storage_write_latency_le_* | Histogram | The entry rate of a topic that the storage write latency is smaller with a given threshold.<br /> Available thresholds: <br /><ul><li>pulsar_storage_write_latency_le_0_5: <= 0.5ms </li><li>pulsar_storage_write_latency_le_1: <= 1ms</li><li>pulsar_storage_write_latency_le_5: <= 5ms</li><li>pulsar_storage_write_latency_le_10: <= 10ms</li><li>pulsar_storage_write_latency_le_20: <= 20ms</li><li>pulsar_storage_write_latenc [...] +| pulsar_entry_size_le_* | Histogram | The entry rate of a topic that the entry size is smaller with a given threshold.<br /> Available thresholds: <br /><ul><li>pulsar_entry_size_le_128: <= 128 bytes </li><li>pulsar_entry_size_le_512: <= 512 bytes</li><li>pulsar_entry_size_le_1_kb: <= 1 KB</li><li>pulsar_entry_size_le_2_kb: <= 2 KB</li><li>pulsar_entry_size_le_4_kb: <= 4 KB</li><li>pulsar_entry_size_le_16_kb: <= 16 KB</li><li>pulsar_entry_size_le_100_kb: [...] +| pulsar_in_bytes_total | Counter | The total number of messages in bytes received for this topic. [...] +| pulsar_in_messages_total | Counter | The total number of messages received for this topic. [...] +| pulsar_out_bytes_total | Counter | The total number of messages in bytes read from this topic. [...] +| pulsar_out_messages_total | Counter | The total number of messages read from this topic. [...] +| pulsar_compaction_removed_event_count | Gauge | The total number of removed events of the compaction. [...] +| pulsar_compaction_succeed_count | Gauge | The total number of successes of the compaction. [...] +| pulsar_compaction_failed_count | Gauge | The total number of failures of the compaction. [...] +| pulsar_compaction_duration_time_in_mills | Gauge | The duration time of the compaction. [...] +| pulsar_compaction_read_throughput | Gauge | The read throughput of the compaction. [...] +| pulsar_compaction_write_throughput | Gauge | The write throughput of the compaction. [...] +| pulsar_compaction_latency_le_* | Histogram | The compaction latency with given quantile. <br /> Available thresholds: <br /><ul><li>pulsar_compaction_latency_le_0_5: <= 0.5ms </li><li>pulsar_compaction_latency_le_1: <= 1ms</li><li>pulsar_compaction_latency_le_5: <= 5ms</li><li>pulsar_compaction_latency_le_10: <= 10ms</li><li>pulsar_compaction_latency_le_20: <= 20ms</li><li>pulsar_compaction_latency_le_50: <= 50ms</li><li>pulsar_compaction_latency_le_100: <= 100 [...] +| pulsar_compaction_compacted_entries_count | Gauge | The total number of the compacted entries. [...] +| pulsar_compaction_compacted_entries_size | Gauge | The total size of the compacted entries. [...] +| pulsar_txn_tb_active_total | Gauge | The number of active transactions on this topic. [...] +| pulsar_txn_tb_aborted_total | Counter | The number of aborted transactions on the topic. [...] +| pulsar_txn_tb_committed_total | Counter | The number of committed transactions on the topic. [...] +| pulsar_delayed_message_index_size_bytes | Gauge | The total memory size allocated by `DelayedDeliveryTracker` of the topic owned by this broker (in bytes). [...] +| pulsar_delayed_message_index_bucket_total | Gauge | The number of delayed message index buckets (immutable buckets + LastMutableBucket ) [...] +| pulsar_delayed_message_index_loaded | Gauge | The total number of delayed message indexes for in the memory. [...] +| pulsar_delayed_message_index_bucket_snapshot_size_bytes | Gauge | The total size of delayed message index bucket snapshot (in bytes). [...] +| pulsar_delayed_message_index_bucket_op_count | Counter | The total number of operation delayed message index bucket snapshots. The `state` label can be `succeed`,`failed`, and`all` (`all` means the total number of all states) and the `type` label can be `create`,`load`,`delete`, and `merge`. [...] +| pulsar_delayed_message_index_bucket_op_latency_ms | Histogram | The latency of delayed message index bucket snapshot operation with a given quantile (threshold). The label`type` label can be `create`,`load`,`delete`, and `merge`<br/>The label `quantile` can be:<ul><li>quantile="50" is operation latency between (0ms, 50ms]</li><li>quantile="100" is operation latency between (50ms, 100ms]</li><li>quantile="500" is operation latency between (100ms, 500ms]</li><li>quantile="1000" is [...] ### Replication metrics @@ -339,16 +344,16 @@ All the cursor acknowledgment state metrics are labeled with the following label - cursor_name: `ledger_name=${pulsar_cursor_name}`. `${pulsar_cursor_name}` is the cursor name. -Name |Type |Description -|---|---|--- -pulsar_ml_cursor_persistLedgerSucceed|Gauge|The number of acknowledgment states that is persistent to a ledger.| -pulsar_ml_cursor_persistLedgerErrors|Gauge|The number of ledger errors occurred when acknowledgment states fail to be persistent to the ledger.| -pulsar_ml_cursor_persistZookeeperSucceed|Gauge|The number of acknowledgment states that is persistent to ZooKeeper. -pulsar_ml_cursor_persistZookeeperErrors|Gauge|The number of ledger errors occurred when acknowledgment states fail to be persistent to ZooKeeper. -pulsar_ml_cursor_nonContiguousDeletedMessagesRange|Gauge|The number of non-contiguous deleted messages ranges. -pulsar_ml_cursor_writeLedgerSize|Gauge|The size of write to ledger. -pulsar_ml_cursor_writeLedgerLogicalSize|Gauge|The size of write to ledger (accounting for without replicas). -pulsar_ml_cursor_readLedgerSize|Gauge|The size of read from ledger. +| Name | Type | Description | +|----------------------------------------------------|-------|------------------------------------------------------------------------------------------------------| +| pulsar_ml_cursor_persistLedgerSucceed | Gauge | The number of acknowledgment states that is persistent to a ledger. | +| pulsar_ml_cursor_persistLedgerErrors | Gauge | The number of ledger errors occurred when acknowledgment states fail to be persistent to the ledger. | +| pulsar_ml_cursor_persistZookeeperSucceed | Gauge | The number of acknowledgment states that is persistent to ZooKeeper. | +| pulsar_ml_cursor_persistZookeeperErrors | Gauge | The number of ledger errors occurred when acknowledgment states fail to be persistent to ZooKeeper. | +| pulsar_ml_cursor_nonContiguousDeletedMessagesRange | Gauge | The number of non-contiguous deleted messages ranges. | +| pulsar_ml_cursor_writeLedgerSize | Gauge | The size of write to ledger. | +| pulsar_ml_cursor_writeLedgerLogicalSize | Gauge | The size of write to ledger (accounting for without replicas). | +| pulsar_ml_cursor_readLedgerSize | Gauge | The size of read from ledger. | ### Subscription metrics @@ -727,9 +732,9 @@ All the bundle assign metrics are labeled with the following labels: - bundle: bundle=${bundle}. ${bundle} is the bundle range on this broker. - metric: metric="assign". -Name | Type | Description -|---|---|--- -pulsar_lb_assign_broker_breakdown_total{result, reason}*|Counter| Assign broker breakdown count grouped by result and reason labels.| +| Name | Type | Description | +|----------------------------------------------------------|---------|--------------------------------------------------------------------| +| pulsar_lb_assign_broker_breakdown_total{result, reason}* | Counter | Assign broker breakdown count grouped by result and reason labels. | ### Service unit state channel metrics @@ -738,16 +743,16 @@ All the service unit state channel metrics are labeled with the following labels - cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name you have configured in the `broker.conf` file. - metric: metric="sunitStateChn". -Name | Type | Description -|---|---|--- -pulsar_sunit_state_chn_owner_lookup_total{result, state}*|Counter|The owner broker lookup counts grouped by the result and state labels. -pulsar_sunit_state_chn_event_publish_ops_total{result, event}*|Counter|The published message count of service unit (e.g., bundle) state changes grouped by the result and event labels -pulsar_sunit_state_chn_subscribe_ops_total{result, event}*|Counter|The subscribed message count of service unit (e.g., bundle) state changes grouped by the result and event labels. -pulsar_sunit_state_chn_inactive_broker_cleanup_ops_total{result}*|Counter|The counts of inactive broker cleanup operations grouped by the result label. -pulsar_sunit_state_chn_orphan_su_cleanup_ops_total*|Counter|The total count of orphan service unit (e.g., bundle) cleanup operations. -pulsar_sunit_state_chn_owned_su_total*|Gauge|The number of owned bundles. -pulsar_sunit_state_chn_su_tombstone_cleanup_ops_total*|Counter|The total count of deleted service units (e.g., bundles) tombstone operations. -pulsar_sunit_state_chn_cleanup_ops_total{result=Failure}*|Counter|The total count of cleanup operation failures. +| Name | Type | Description | +|-------------------------------------------------------------------|---------|-------------------------------------------------------------------------------------------------------------------| +| pulsar_sunit_state_chn_owner_lookup_total{result, state}* | Counter | The owner broker lookup counts grouped by the result and state labels. | +| pulsar_sunit_state_chn_event_publish_ops_total{result, event}* | Counter | The published message count of service unit (e.g., bundle) state changes grouped by the result and event labels | +| pulsar_sunit_state_chn_subscribe_ops_total{result, event}* | Counter | The subscribed message count of service unit (e.g., bundle) state changes grouped by the result and event labels. | +| pulsar_sunit_state_chn_inactive_broker_cleanup_ops_total{result}* | Counter | The counts of inactive broker cleanup operations grouped by the result label. | +| pulsar_sunit_state_chn_orphan_su_cleanup_ops_total* | Counter | The total count of orphan service unit (e.g., bundle) cleanup operations. | +| pulsar_sunit_state_chn_owned_su_total* | Gauge | The number of owned bundles. | +| pulsar_sunit_state_chn_su_tombstone_cleanup_ops_total* | Counter | The total count of deleted service units (e.g., bundles) tombstone operations. | +| pulsar_sunit_state_chn_cleanup_ops_total{result=Failure}* | Counter | The total count of cleanup operation failures. | ## Pulsar Functions @@ -756,20 +761,20 @@ All the Pulsar Functions metrics are labeled with the following labels: - *cluster*: `cluster=${pulsar_cluster}`. `${pulsar_cluster}` is the cluster name that you have configured in the `broker.conf` file. - *namespace*: `namespace=${pulsar_namespace}`. `${pulsar_namespace}` is the namespace name. -| Name | Type | Description | -|---|---|---| -| pulsar_function_processed_successfully_total | Counter | The total number of messages processed successfully. | +| Name | Type | Description | +|---------------------------------------------------|---------|---------------------------------------------------------------------------| +| pulsar_function_processed_successfully_total | Counter | The total number of messages processed successfully. | | pulsar_function_processed_successfully_1min_total | Counter | The total number of messages processed successfully in the last 1 minute. | -| pulsar_function_system_exceptions_total | Counter | The total number of system exceptions. | -| pulsar_function_system_exceptions_1min_total | Counter | The total number of system exceptions in the last 1 minute. | -| pulsar_function_user_exceptions_total | Counter | The total number of user exceptions. | -| pulsar_function_user_exceptions_1min_total | Counter | The total number of user exceptions in the last 1 minute. | -| pulsar_function_process_latency_ms | Summary | The process latency in milliseconds. | -| pulsar_function_process_latency_ms_1min | Summary | The process latency in milliseconds in the last 1 minute. | -| pulsar_function_last_invocation | Gauge | The timestamp of the last invocation of the function. | -| pulsar_function_received_total | Counter | The total number of messages received from source. | -| pulsar_function_received_1min_total | Counter | The total number of messages received from source in the last 1 minute. | -pulsar_function_user_metric_ | Summary|The user-defined metrics. +| pulsar_function_system_exceptions_total | Counter | The total number of system exceptions. | +| pulsar_function_system_exceptions_1min_total | Counter | The total number of system exceptions in the last 1 minute. | +| pulsar_function_user_exceptions_total | Counter | The total number of user exceptions. | +| pulsar_function_user_exceptions_1min_total | Counter | The total number of user exceptions in the last 1 minute. | +| pulsar_function_process_latency_ms | Summary | The process latency in milliseconds. | +| pulsar_function_process_latency_ms_1min | Summary | The process latency in milliseconds in the last 1 minute. | +| pulsar_function_last_invocation | Gauge | The timestamp of the last invocation of the function. | +| pulsar_function_received_total | Counter | The total number of messages received from source. | +| pulsar_function_received_1min_total | Counter | The total number of messages received from source in the last 1 minute. | +| pulsar_function_user_metric_ | Summary | The user-defined metrics. | ## Connectors @@ -782,37 +787,37 @@ Connector metrics contain **source** metrics and **sink** metrics. - **Source** metrics - | Name | Type | Description | - |---|---|---| - pulsar_source_written_total|Counter|The total number of records written to a Pulsar topic. - pulsar_source_written_1min_total|Counter|The total number of records written to a Pulsar topic in the last 1 minute. - pulsar_source_received_total|Counter|The total number of records received from source. - pulsar_source_received_1min_total|Counter|The total number of records received from source in the last 1 minute. - pulsar_source_last_invocation|Gauge|The timestamp of the last invocation of the source. - pulsar_source_source_exception|Gauge|The exception from a source. - pulsar_source_source_exceptions_total|Counter|The total number of source exceptions. - pulsar_source_source_exceptions_1min_total |Counter|The total number of source exceptions in the last 1 minute. - pulsar_source_system_exception|Gauge|The exception from system code. - pulsar_source_system_exceptions_total|Counter|The total number of system exceptions. - pulsar_source_system_exceptions_1min_total|Counter|The total number of system exceptions in the last 1 minute. - pulsar_source_user_metric_ | Summary|The user-defined metrics. + | Name | Type | Description | + |--------------------------------------------|---------|-----------------------------------------------------------------------------| + | pulsar_source_written_total | Counter | The total number of records written to a Pulsar topic. | + | pulsar_source_written_1min_total | Counter | The total number of records written to a Pulsar topic in the last 1 minute. | + | pulsar_source_received_total | Counter | The total number of records received from source. | + | pulsar_source_received_1min_total | Counter | The total number of records received from source in the last 1 minute. | + | pulsar_source_last_invocation | Gauge | The timestamp of the last invocation of the source. | + | pulsar_source_source_exception | Gauge | The exception from a source. | + | pulsar_source_source_exceptions_total | Counter | The total number of source exceptions. | + | pulsar_source_source_exceptions_1min_total | Counter | The total number of source exceptions in the last 1 minute. | + | pulsar_source_system_exception | Gauge | The exception from system code. | + | pulsar_source_system_exceptions_total | Counter | The total number of system exceptions. | + | pulsar_source_system_exceptions_1min_total | Counter | The total number of system exceptions in the last 1 minute. | + | pulsar_source_user_metric_ | Summary | The user-defined metrics. | - **Sink** metrics - | Name | Type | Description | - |---|---|---| - pulsar_sink_written_total|Counter| The total number of records processed by a sink. - pulsar_sink_written_1min_total|Counter| The total number of records processed by a sink in the last 1 minute. - pulsar_sink_received_1min_total|Counter| The total number of messages that a sink has received from Pulsar topics in the last 1 minute. - pulsar_sink_received_total|Counter| The total number of records that a sink has received from Pulsar topics. - pulsar_sink_last_invocation|Gauge|The timestamp of the last invocation of the sink. - pulsar_sink_sink_exception|Gauge|The exception from a sink. - pulsar_sink_sink_exceptions_total|Counter|The total number of sink exceptions. - pulsar_sink_sink_exceptions_1min_total |Counter|The total number of sink exceptions in the last 1 minute. - pulsar_sink_system_exception|Gauge|The exception from system code. - pulsar_sink_system_exceptions_total|Counter|The total number of system exceptions. - pulsar_sink_system_exceptions_1min_total|Counter|The total number of system exceptions in the last 1 minute. - pulsar_sink_user_metric_ | Summary|The user-defined metrics. + | Name | Type | Description | + |------------------------------------------|---------|------------------------------------------------------------------------------------------------| + | pulsar_sink_written_total | Counter | The total number of records processed by a sink. | + | pulsar_sink_written_1min_total | Counter | The total number of records processed by a sink in the last 1 minute. | + | pulsar_sink_received_1min_total | Counter | The total number of messages that a sink has received from Pulsar topics in the last 1 minute. | + | pulsar_sink_received_total | Counter | The total number of records that a sink has received from Pulsar topics. | + | pulsar_sink_last_invocation | Gauge | The timestamp of the last invocation of the sink. | + | pulsar_sink_sink_exception | Gauge | The exception from a sink. | + | pulsar_sink_sink_exceptions_total | Counter | The total number of sink exceptions. | + | pulsar_sink_sink_exceptions_1min_total | Counter | The total number of sink exceptions in the last 1 minute. | + | pulsar_sink_system_exception | Gauge | The exception from system code. | + | pulsar_sink_system_exceptions_total | Counter | The total number of system exceptions. | + | pulsar_sink_system_exceptions_1min_total | Counter | The total number of system exceptions in the last 1 minute. | + | pulsar_sink_user_metric_ | Summary | The user-defined metrics. | ## Proxy diff --git a/docs/reference-terminology.md b/docs/reference-terminology.md index ee6754773753..74146ea36704 100644 --- a/docs/reference-terminology.md +++ b/docs/reference-terminology.md @@ -18,6 +18,12 @@ Pulsar is a distributed messaging system originally created by Yahoo but now und Namespace bundle is a virtual group of [topics](concepts-messaging.md#topics) that belong to the same [namespace](concepts-multi-tenancy.md#namespaces). A namespace bundle is defined as a range between two 32-bit hashes, such as 0x00000000 and 0xffffffff. +### Namespace + +Namespace is a virtual grouping of [topics](concepts-messaging.md#topics), under a specific +[tenant](concepts-multi-tenancy.md#tenants). A namespace is defined +by a string name, such as `my-tenant/my-namespace`. + ### Pub-Sub Pub-sub is a messaging pattern in which [producer](concepts-clients.md#producer) processes publish messages on [topics](concepts-messaging.md#topics) that @@ -77,6 +83,11 @@ the topic needs to be routed). Dispatcher is an asynchronous TCP server used for all data transfers in and out of a Pulsar [broker](concepts-architecture-overview.md#brokers). The Pulsar dispatcher uses a custom binary protocol for all communications. +### Broker + +Broker is a Pulsar server that receives, acknowledges, and delivers messages to consumers. +A Pulsar cluster can have one or more brokers. + ## Storage ### Bookie
