This is an automated email from the ASF dual-hosted git repository.
wusheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/skywalking.git
The following commit(s) were added to refs/heads/master by this push:
new ced3f22708 Feature pulsar monitoring (#11339)
ced3f22708 is described below
commit ced3f227088090bd32f3bd70025463f1f40893a4
Author: Xiangying Meng <55571188+liangyepianz...@users.noreply.github.com>
AuthorDate: Tue Oct 31 21:04:16 2023 +0800
Feature pulsar monitoring (#11339)
---
.github/workflows/skywalking.yaml | 2 +
docs/en/changes/changes.md | 1 +
.../setup/backend/backend-bookkeeper-monitoring.md | 61 +++
docs/en/setup/backend/backend-pulsar-monitoring.md | 67 ++++
docs/menu.yml | 4 +
.../skywalking/oap/server/core/analysis/Layer.java | 14 +-
.../ui/template/UITemplateInitializer.java | 2 +
.../src/main/resources/application.yml | 2 +-
.../otel-rules/bookkeeper/bookkeeper-cluster.yaml | 62 +++
.../otel-rules/bookkeeper/bookkeeper-node.yaml | 90 +++++
.../resources/otel-rules/pulsar/pulsar-broker.yaml | 76 ++++
.../otel-rules/pulsar/pulsar-cluster.yaml | 68 ++++
.../bookkeeper/bookkeeper-cluster.json | 393 +++++++++++++++++++
.../bookkeeper/bookkeeper-node.json | 366 +++++++++++++++++
.../bookkeeper/bookkeeper-root.json | 51 +++
.../resources/ui-initialized-templates/menu.yaml | 10 +
.../pulsar/pulsar-broker.json | 422 ++++++++++++++++++++
.../pulsar/pulsar-cluster.json | 435 +++++++++++++++++++++
.../pulsar/pulsar-root.json | 51 +++
test/e2e-v2/cases/pulsar/docker-compose.yml | 137 +++++++
test/e2e-v2/cases/pulsar/e2e.yaml | 37 ++
.../cases/pulsar/expected/bookie_instance.yml | 22 ++
.../cases/pulsar/expected/broker_instance.yml | 22 ++
.../metrics-has-value-instance-bookie-label.yml | 33 ++
.../metrics-has-value-instance-broker-label.yml | 33 ++
.../cases/pulsar/expected/metrics-has-value.yml | 31 ++
test/e2e-v2/cases/pulsar/expected/service.yml | 31 ++
.../e2e-v2/cases/pulsar/otel-collector-config.yaml | 58 +++
test/e2e-v2/cases/pulsar/pulsar-cases.yaml | 129 ++++++
29 files changed, 2708 insertions(+), 2 deletions(-)
diff --git a/.github/workflows/skywalking.yaml
b/.github/workflows/skywalking.yaml
index bfb1b97bcb..96b3a0736b 100644
--- a/.github/workflows/skywalking.yaml
+++ b/.github/workflows/skywalking.yaml
@@ -641,6 +641,8 @@ jobs:
config: test/e2e-v2/cases/kafka/kafka-monitoring/e2e.yaml
- name: MQE Service
config: test/e2e-v2/cases/mqe/e2e.yaml
+ - name: Pulsar and BookKeeper
+ config: test/e2e-v2/cases/pulsar/e2e.yaml
- name: UI Menu BanyanDB
config: test/e2e-v2/cases/menu/banyandb/e2e.yaml
diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md
index e2e7f614c1..4ba8bb4417 100644
--- a/docs/en/changes/changes.md
+++ b/docs/en/changes/changes.md
@@ -13,6 +13,7 @@
* ElasticSearchClient: Add `deleteById` API.
* Fix Custom alarm rules are overwritten by 'resource/alarm-settings.yml'
* Support Kafka Monitoring.
+* Support Pulsar server and BookKeeper server Monitoring.
* [Breaking Change] Elasticsearch storage merge all management data indices
into one index `management`,
including `ui_template，ui_menu，continuous_profiling_policy`.
* Add a release mechanism for alarm windows when it is expired in case of OOM.
diff --git a/docs/en/setup/backend/backend-bookkeeper-monitoring.md
b/docs/en/setup/backend/backend-bookkeeper-monitoring.md
new file mode 100644
index 0000000000..e94e347481
--- /dev/null
+++ b/docs/en/setup/backend/backend-bookkeeper-monitoring.md
@@ -0,0 +1,61 @@
+# BookKeeper monitoring
+
+SkyWalking leverages OpenTelemetry Collector to collect metrics data from the
BookKeeper and leverages OpenTelemetry Collector to transfer the metrics to
+[OpenTelemetry receiver](opentelemetry-receiver.md) and into the [Meter
System](./../../concepts-and-designs/meter.md).
+Kafka entity as a `Service` in OAP and on the `Layer: BOOKKEEPER.
+
+## Data flow
+
+1. BookKeeper exposes metrics through Prometheus endpoint.
+2. OpenTelemetry Collector fetches metrics from BookKeeper cluster via
Prometheus Receiver and pushes metrics to SkyWalking OAP Server via
OpenTelemetry gRPC exporter.
+3. The SkyWalking OAP Server parses the expression with
[MAL](../../concepts-and-designs/mal.md) to
+ filter/calculate/aggregate and store the results.`
+
+## Setup
+
+1. Set up [BookKeeper
Cluster](https://bookkeeper.apache.org/docs/deployment/manual).
+2. Set up [OpenTelemetry
Collector](https://opentelemetry.io/docs/collector/getting-started/#kubernetes).
The example
+ for OpenTelemetry Collector configuration, refer
+ to [here](../../../../test/e2e-v2/cases/pulsar/otel-collector-config.yaml).
+3. Config SkyWalking [OpenTelemetry receiver](opentelemetry-receiver.md).
+
+## BookKeeper Monitoring
+
+Bookkeeper monitoring provides multidimensional metrics monitoring of
BookKeeper cluster as `Layer: BOOKKEEPER` `Service` in
+the OAP. In each cluster, the nodes are represented as `Instance`.
+
+### BookKeeper Cluster Supported Metrics
+
+| Monitoring Panel | Metric Name
| Description | Data
Source |
+|--------------------------------|------------------------------------------------------------------|---------------------------------------------------|---------------------|
+| Bookie Ledgers Count | meter_bookkeeper_bookie_ledgers_count
| The number of the bookie ledgers. |
Bookkeeper Cluster |
+| Bookie Ledger Writable Dirs |
meter_bookkeeper_bookie_ledger_writable_dirs | The number
of writable directories in the bookie. | Bookkeeper Cluster |
+| Bookie Ledger Dir Usage |
meter_bookkeeper_bookie_ledger_dir_data_bookkeeper_ledgers_usage | The number
of successfully created connections. | Bookkeeper Cluster |
+| Bookie Entries Count | meter_bookkeeper_bookie_entries_count
| The number of the bookie write entries. |
Bookkeeper Cluster |
+| Bookie Write Cache Size | meter_bookkeeper_bookie_write_cache_size
| The size of the bookie write cache (MB). |
Bookkeeper Cluster |
+| Bookie Write Cache Entry Count | meter_bookkeeper_bookie_write_cache_count
| The entry count in the bookie write cache. |
Bookkeeper Cluster |
+| Bookie Read Cache Size | meter_bookkeeper_bookie_read_cache_size
| The size of the bookie read cache (MB). |
Bookkeeper Cluster |
+| Bookie Read Cache Entry Count | meter_bookkeeper_bookie_read_cache_count
| The entry count in the bookie read cache. |
Bookkeeper Cluster |
+| Bookie Read Rate | meter_bookkeeper_bookie_read_rate
| The bookie read rate (bytes/s). |
Bookkeeper Cluster |
+| Bookie Write Rate | meter_bookkeeper_bookie_write_rate
| The bookie write rate (bytes/s). |
Bookkeeper Cluster |
+
+### BookKeeper Node Supported Metrics
+
+| Monitoring Panel | Metric Name
| Description
| Data Source |
+|-------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------|--------------------|
+| JVM Memory Pool Used | meter_bookkeeper_node_jvm_memory_pool_used
| The usage of
the broker jvm memory pool. | Bookkeeper Bookie |
+| JVM Memory | meter_bookkeeper_node_jvm_memory_used <br />
meter_bookkeeper_node_jvm_memory_committed <br />
meter_bookkeeper_node_jvm_memory_init
| The usage of the broker jvm memory. |
Bookkeeper Bookie |
+| JVM Threads | meter_bookkeeper_node_jvm_threads_current
<br /> meter_bookkeeper_node_jvm_threads_daemon <br />
meter_bookkeeper_node_jvm_threads_peak <br />
meter_bookkeeper_node_jvm_threads_deadlocked | The count of the jvm threads.
| Bookkeeper Bookie |
+| GC Time |
meter_bookkeeper_node_jvm_gc_collection_seconds_sum
| Time spent in a given JVM garbage collector in
seconds. | Bookkeeper Bookie |
+| GC Count |
meter_bookkeeper_node_jvm_gc_collection_seconds_count
| The count of a given JVM garbage.
| Bookkeeper Bookie |
+| Thread Executor Completed |
meter_bookkeeper_node_thread_executor_completed
| The count of the executor thread.
| Bookkeeper Bookie |
+| Thread Executor Tasks |
meter_bookkeeper_node_thread_executor_tasks_completed <br />
meter_bookkeeper_node_thread_executor_tasks_rejected <br />
meter_bookkeeper_node_thread_executor_tasks_failed | The count
of the executor tasks. | Bookkeeper Bookie |
+| Pooled Threads | meter_bookkeeper_node_high_priority_threads
<br /> meter_bookkeeper_node_read_thread_pool_threads
| The count of
the pooled thread. | Bookkeeper Bookie |
+| Pooled Threads Max Queue Size |
meter_bookkeeper_node_high_priority_thread_max_queue_size <br />
meter_bookkeeper_node_read_thread_pool_max_queue_size
| The count of the pooled threads max
queue size. | Bookkeeper Bookie |
+
+## Customizations
+
+You can customize your own metrics/expression/dashboard panel.
+The metrics definition and expression rules are found
+in `otel-rules/bookkeeper/bookkeeper-cluster.yaml,
otel-rules/bookkeeper/bookkeeper-node.yaml`.
+The RabbitMQ dashboard panel configurations are found in
`/config/ui-initialized-templates/bookkeeper`.
diff --git a/docs/en/setup/backend/backend-pulsar-monitoring.md
b/docs/en/setup/backend/backend-pulsar-monitoring.md
new file mode 100644
index 0000000000..ced0adee38
--- /dev/null
+++ b/docs/en/setup/backend/backend-pulsar-monitoring.md
@@ -0,0 +1,67 @@
+# Pulsar monitoring
+
+SkyWalking leverages OpenTelemetry Collector to collect metrics data in
Prometheus format from the Pulsar and transfer the metrics to
+[OpenTelemetry receiver](opentelemetry-receiver.md) and into the [Meter
System](./../../concepts-and-designs/meter.md).
+Kafka entity as a `Service` in OAP and on the `Layer: PULSAR.
+
+## Data flow
+
+1. Pulsar exposes metrics through Prometheus endpoint.
+2. OpenTelemetry Collector fetches metrics from Pulsar cluster via Prometheus
Receiver and pushes metrics to SkyWalking OAP Server via OpenTelemetry gRPC
exporter.
+3. The SkyWalking OAP Server parses the expression with
[MAL](../../concepts-and-designs/mal.md) to
+ filter/calculate/aggregate and store the results.`
+
+## Setup
+
+1. Set up [Pulsar
Cluster](https://pulsar.apache.org/docs/3.1.x/getting-started-docker-compose/).
(Pulsar cluster includes pulsar broker cluster and Bookkeeper bookie cluster.)
+2. Set up [OpenTelemetry
Collector](https://opentelemetry.io/docs/collector/getting-started/#kubernetes).
The example
+ for OpenTelemetry Collector configuration, refer
+ to [here](../../../../test/e2e-v2/cases/pulsar/otel-collector-config.yaml).
+3. Config SkyWalking [OpenTelemetry receiver](opentelemetry-receiver.md).
+
+## Pulsar Monitoring
+
+Pulsar monitoring provides multidimensional metrics monitoring of Pulsar
cluster as `Layer: PULSAR` `Service` in
+the OAP. In each cluster, the nodes are represented as `Instance`.
+
+### Pulsar Cluster Supported Metrics
+
+| Monitoring Panel | Metric Name |
Description
| Data Source |
+|----------------------|--------------------------------------------|--------------------------------------------------------------------------------------------------------|----------------|
+| Total Topics | meter_pulsar_total_topics | The
number of Pulsar topics in this cluster.
| Pulsar Cluster |
+| Total Subscriptions | meter_pulsar_total_subscriptions | The
number of Pulsar subscriptions in this cluster.
| Pulsar Cluster |
+| Total Producers | meter_pulsar_total_producers | The
number of active producers connected to this cluster.
| Pulsar Cluster |
+| Total Consumers | meter_pulsar_total_consumers | The
number of active consumers connected to this cluster.
| Pulsar Cluster |
+| Message Rate In | meter_pulsar_message_rate_in | The
total message rate coming into this cluster (message per second).
| Pulsar Cluster |
+| Message Rate Out | meter_pulsar_message_rate_out | The
total message rate going out from this cluster (message per second).
| Pulsar Cluster |
+| Throughput In | meter_pulsar_throughput_in | The
total throughput coming into this cluster (byte per second).
| Pulsar Cluster |
+| Throughput Out | meter_pulsar_throughput_out | The
total throughput going out from this cluster (byte per second).
| Pulsar Cluster |
+| Storage Size | meter_pulsar_storage_size | The
total storage size of all topics in this broker (in bytes).
| Pulsar Cluster |
+| Storage Logical Size | meter_pulsar_storage_logical_size | The
storage size of all topics in this broker without replicas (in bytes).
| Pulsar Cluster |
+| Storage Write Rate | meter_pulsar_storage_write_rate | The
total message batches (entries) written to the storage for this broker (message
batch per second). | Pulsar Cluster |
+| Storage Read Rate | meter_pulsar_storage_read_rate | The
total message batches (entries) read from the storage for this broker (message
batch per second). | Pulsar Cluster |
+
+
+### Pulsar Node Supported Metrics
+
+
+| Monitoring Panel | Metric Name
| Description
| Data Source |
+|---------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------|----------------|
+| Active Connections | meter_pulsar_broker_active_connections
| The number of active
connections. | Pulsar Broker |
+| Total Connections | meter_pulsar_broker_total_connections
| The total number of
connections. | Pulsar Broker |
+| Connection Create Success Count |
meter_pulsar_broker_connection_create_success_count
| The number of successfully created connections. |
Pulsar Broker |
+| Connection Create Fail Count |
meter_pulsar_broker_connection_create_fail_count
| The number of failed connections. |
Pulsar Broker |
+| Connection Closed Total Count |
meter_pulsar_broker_connection_closed_total_count
| The total number of closed connections. |
Pulsar Broker |
+| JVM Buffer Pool Used |
meter_pulsar_broker_jvm_buffer_pool_used_bytes
| The usage of jvm buffer pool. |
Pulsar Broker |
+| JVM Memory Pool Used | meter_pulsar_broker_jvm_memory_pool_used
| The usage of jvm
memory pool. | Pulsar Broker |
+| JVM Memory | meter_pulsar_broker_jvm_memory_init <br />
meter_pulsar_broker_jvm_memory_used <br />
meter_pulsar_broker_jvm_memory_committed
| The usage of jvm memory. |
Pulsar Broker |
+| JVM Threads | meter_pulsar_broker_jvm_threads_current
<br /> meter_pulsar_broker_jvm_threads_daemon <br />
meter_pulsar_broker_jvm_threads_peak <br />
meter_pulsar_broker_jvm_threads_deadlocked | The usage of jvm threads.
| Pulsar Broker |
+| GC Time |
meter_pulsar_broker_jvm_gc_collection_seconds_sum
| Time spent in a given JVM garbage collector in seconds. |
Pulsar Broker |
+| GC Count |
meter_pulsar_broker_jvm_gc_collection_seconds_count
| The count of a given JVM garbage collector. |
Pulsar Broker |
+
+## Customizations
+
+You can customize your own metrics/expression/dashboard panel.
+The metrics definition and expression rules are found
+in `otel-rules/pulsar/pulsar-cluster.yaml,
otel-rules/pulsar/pulsar-broker.yaml`.
+The RabbitMQ dashboard panel configurations are found in
`ui-initialized-templates/pulsar`.
diff --git a/docs/menu.yml b/docs/menu.yml
index 4e5203a8cc..dc40ecd939 100644
--- a/docs/menu.yml
+++ b/docs/menu.yml
@@ -259,12 +259,16 @@ catalog:
path: "/en/setup/backend/backend-elasticsearch-monitoring"
- name: "MongoDB"
path: "/en/setup/backend/backend-mongodb-monitoring"
+ - name: "BookKeeper"
+ path: "/en/setup/backend/backend-bookkeeper-monitoring"
- name: "MQ Monitoring"
catalog:
- name: "RabbitMQ"
path: "/en/setup/backend/backend-rabbitmq-monitoring"
- name: "Kafka"
path: "/en/setup/backend/backend-kafka-monitoring"
+ - name: "Pulsar"
+ path: "/en/setup/backend/backend-pulsar-monitoring"
- name: "Self Observability"
catalog:
- name: "OAP self telemetry"
diff --git
a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java
b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java
index ab5e54d18f..46cf71f52a 100644
---
a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java
+++
b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java
@@ -193,7 +193,19 @@ public enum Layer {
/**
* Kafka is a distributed streaming platform that is used publish and
subscribe to streams of records.
*/
- KAFKA(31, true);
+ KAFKA(31, true),
+
+ /**
+ * Pulsar is a distributed pub-sub messaging platform that provides
high-performance, durable messaging.
+ * It is used to publish and subscribe to streams of records.
+ * Pulsar supports scalable and fault-tolerant messaging, making it
suitable for use in distributed systems.
+ */
+ PULSAR(32, true),
+
+ /**
+ * A scalable, fault-tolerant, and low-latency storage service optimized
for real-time workloads.
+ */
+ BOOKKEEPER(33, true);
private final int value;
/**
diff --git
a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/management/ui/template/UITemplateInitializer.java
b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/management/ui/template/UITemplateInitializer.java
index 0ae525b6a3..686619c428 100644
---
a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/management/ui/template/UITemplateInitializer.java
+++
b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/management/ui/template/UITemplateInitializer.java
@@ -70,6 +70,8 @@ public class UITemplateInitializer {
Layer.RABBITMQ.name(),
Layer.MONGODB.name(),
Layer.KAFKA.name(),
+ Layer.PULSAR.name(),
+ Layer.BOOKKEEPER.name(),
"custom"
};
private final UITemplateManagementService uiTemplateManagementService;
diff --git a/oap-server/server-starter/src/main/resources/application.yml
b/oap-server/server-starter/src/main/resources/application.yml
index a38a532311..db24ed625c 100644
--- a/oap-server/server-starter/src/main/resources/application.yml
+++ b/oap-server/server-starter/src/main/resources/application.yml
@@ -340,7 +340,7 @@ receiver-otel:
selector: ${SW_OTEL_RECEIVER:default}
default:
enabledHandlers:
${SW_OTEL_RECEIVER_ENABLED_HANDLERS:"otlp-metrics,otlp-logs"}
- enabledOtelMetricsRules:
${SW_OTEL_RECEIVER_ENABLED_OTEL_METRICS_RULES:"apisix,k8s/*,istio-controlplane,vm,mysql/*,postgresql/*,oap,aws-eks/*,windows,aws-s3/*,aws-dynamodb/*,aws-gateway/*,redis/*,elasticsearch/*,rabbitmq/*,mongodb/*,kafka/*"}
+ enabledOtelMetricsRules:
${SW_OTEL_RECEIVER_ENABLED_OTEL_METRICS_RULES:"apisix,k8s/*,istio-controlplane,vm,mysql/*,postgresql/*,oap,aws-eks/*,windows,aws-s3/*,aws-dynamodb/*,aws-gateway/*,redis/*,elasticsearch/*,rabbitmq/*,mongodb/*,kafka/*,pulsar/*,bookkeeper/*"}
receiver-zipkin:
selector: ${SW_RECEIVER_ZIPKIN:-}
diff --git
a/oap-server/server-starter/src/main/resources/otel-rules/bookkeeper/bookkeeper-cluster.yaml
b/oap-server/server-starter/src/main/resources/otel-rules/bookkeeper/bookkeeper-cluster.yaml
new file mode 100644
index 0000000000..6345773c20
--- /dev/null
+++
b/oap-server/server-starter/src/main/resources/otel-rules/bookkeeper/bookkeeper-cluster.yaml
@@ -0,0 +1,62 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This will parse a textual representation of a duration. The formats
+# accepted are based on the ISO-8601 duration format {@code PnDTnHnMn.nS}
+# with days considered to be exactly 24 hours.
+# <p>
+# Examples:
+# <pre>
+# "PT20.345S" -- parses as "20.345 seconds"
+# "PT15M" -- parses as "15 minutes" (where a minute is 60 seconds)
+# "PT10H" -- parses as "10 hours" (where an hour is 3600 seconds)
+# "P2D" -- parses as "2 days" (where a day is 24 hours or 86400
seconds)
+# "P2DT3H4M" -- parses as "2 days, 3 hours and 4 minutes"
+# "P-6H3M" -- parses as "-6 hours and +3 minutes"
+# "-P6H3M" -- parses as "-6 hours and -3 minutes"
+# "-P-6H+3M" -- parses as "+6 hours and -3 minutes"
+# </pre>
+
+filter: "{ tags -> tags.job_name == 'bookkeeper-monitoring' }" # The
OpenTelemetry job name
+expSuffix: tag({tags -> tags.cluster = 'bookkeeper::' +
tags.cluster}).service(['cluster'], Layer.BOOKKEEPER)
+metricPrefix: meter_bookkeeper
+
+# Metrics Rules
+metricsRules:
+
+ # storage metrics
+ - name: bookie_ledgers_count
+ exp: bookie_ledgers_count.sum(['cluster', 'node'])
+ - name: bookie_ledger_writable_dirs
+ exp: bookie_ledger_writable_dirs.sum(['cluster', 'node'])
+ - name: bookie_ledger_dir_data_bookkeeper_ledgers_usage
+ exp: bookie_ledger_dir_data_bookkeeper_ledgers_usage.sum(['cluster',
'node'])
+ - name: bookie_entries_count
+ exp: bookie_entries_count.sum(['cluster', 'node'])
+
+ - name: bookie_write_cache_size
+ exp: bookie_write_cache_size.sum(['cluster', 'node'])
+ - name: bookie_write_cache_count
+ exp: bookie_write_cache_count.sum(['cluster', 'node'])
+
+ - name: bookie_read_cache_size
+ exp: bookie_read_cache_size.sum(['cluster', 'node'])
+ - name: bookie_read_cache_count
+ exp: bookie_read_cache_count.sum(['cluster', 'node'])
+
+ - name: bookie_write_rate
+ exp: bookie_WRITE_BYTES.sum(['cluster', 'node']).rate('PT1M')
+ - name: bookie_read_rate
+ exp: bookie_READ_BYTES.sum(['cluster', 'node']).rate('PT1M')
diff --git
a/oap-server/server-starter/src/main/resources/otel-rules/bookkeeper/bookkeeper-node.yaml
b/oap-server/server-starter/src/main/resources/otel-rules/bookkeeper/bookkeeper-node.yaml
new file mode 100644
index 0000000000..03d3931479
--- /dev/null
+++
b/oap-server/server-starter/src/main/resources/otel-rules/bookkeeper/bookkeeper-node.yaml
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This will parse a textual representation of a duration. The formats
+# accepted are based on the ISO-8601 duration format {@code PnDTnHnMn.nS}
+# with days considered to be exactly 24 hours.
+# <p>
+# Examples:
+# <pre>
+# "PT20.345S" -- parses as "20.345 seconds"
+# "PT15M" -- parses as "15 minutes" (where a minute is 60 seconds)
+# "PT10H" -- parses as "10 hours" (where an hour is 3600 seconds)
+# "P2D" -- parses as "2 days" (where a day is 24 hours or 86400
seconds)
+# "P2DT3H4M" -- parses as "2 days, 3 hours and 4 minutes"
+# "P-6H3M" -- parses as "-6 hours and +3 minutes"
+# "-P6H3M" -- parses as "-6 hours and -3 minutes"
+# "-P-6H+3M" -- parses as "+6 hours and -3 minutes"
+# </pre>
+filter: "{ tags -> tags.job_name == 'bookkeeper-monitoring' }" # The
OpenTelemetry job name
+expSuffix: tag({tags -> tags.cluster = 'bookkeeper::' +
tags.cluster}).instance(['cluster'], ['node'], Layer.BOOKKEEPER)
+metricPrefix: meter_bookkeeper_node
+
+# Metrics Rules
+metricsRules:
+ # thread executor metrics
+ - name: thread_executor_completed
+ exp: bookkeeper_server_thread_executor_completed.sum(['cluster', 'node'])
+ - name: thread_executor_tasks_completed
+ exp: bookkeeper_server_thread_executor_tasks_completed.sum(['cluster',
'node'])
+ - name: thread_executor_tasks_rejected
+ exp: bookkeeper_server_thread_executor_tasks_rejected.sum(['cluster',
'node'])
+ - name: thread_executor_tasks_failed
+ exp: bookkeeper_server_thread_executor_tasks_failed.sum(['cluster',
'node'])
+
+ - name: high_priority_threads
+ exp: bookkeeper_server_BookieHighPriorityThread_threads.sum(['cluster',
'node'])
+ - name: read_thread_pool_threads
+ exp: bookkeeper_server_BookieReadThreadPool_threads.sum(['cluster',
'node'])
+ - name: high_priority_thread_max_queue_size
+ exp:
bookkeeper_server_BookieHighPriorityThread_max_queue_size.sum(['cluster',
'node'])
+ - name: read_thread_pool_max_queue_size
+ exp: bookkeeper_server_BookieReadThreadPool_max_queue_size.sum(['cluster',
'node'])
+
+ # JVM Metrics
+ - name: jvm_memory_used
+ exp: jvm_memory_bytes_used.sum(['cluster', 'node'])
+ - name: jvm_memory_committed
+ exp: jvm_memory_bytes_committed.sum(['cluster', 'node'])
+ - name: jvm_memory_init
+ exp: jvm_memory_bytes_init.sum(['cluster', 'node'])
+
+ - name: jvm_memory_pool_used
+ exp: jvm_memory_pool_bytes_used.sum(['cluster', 'node', 'pool'])
+ - name: jvm_memory_pool_committed
+ exp: jvm_memory_pool_bytes_committed.sum(['cluster', 'node', 'pool'])
+ - name: jvm_memory_pool_init
+ exp: jvm_memory_pool_bytes_init.sum(['cluster', 'node', 'pool'])
+
+ - name: jvm_buffer_pool_used_bytes
+ exp: jvm_buffer_pool_used_bytes.sum(['cluster', 'node', 'pool'])
+ - name: jvm_buffer_pool_capacity_bytes
+ exp: jvm_buffer_pool_capacity_bytes.sum(['cluster', 'node', 'pool'])
+ - name: jvm_buffer_pool_used_buffers
+ exp: jvm_buffer_pool_used_buffers.sum(['cluster', 'node', 'pool'])
+
+ - name: jvm_gc_collection_seconds_count
+ exp: jvm_gc_collection_seconds_count.sum(['cluster', 'node', 'gc'])
+ - name: jvm_gc_collection_seconds_sum
+ exp: jvm_gc_collection_seconds_sum.sum(['cluster', 'node', 'gc'])
+
+ - name: jvm_threads_current
+ exp: jvm_threads_current.sum(['cluster', 'node'])
+ - name: jvm_threads_daemon
+ exp: jvm_threads_daemon.sum(['cluster', 'node'])
+ - name: jvm_threads_peak
+ exp: jvm_threads_peak.sum(['cluster', 'node'])
+ - name: jvm_threads_deadlocked
+ exp: jvm_threads_deadlocked.sum(['cluster', 'node'])
diff --git
a/oap-server/server-starter/src/main/resources/otel-rules/pulsar/pulsar-broker.yaml
b/oap-server/server-starter/src/main/resources/otel-rules/pulsar/pulsar-broker.yaml
new file mode 100644
index 0000000000..038c253c9d
--- /dev/null
+++
b/oap-server/server-starter/src/main/resources/otel-rules/pulsar/pulsar-broker.yaml
@@ -0,0 +1,76 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This will parse a textual representation of a duration. The formats
+# accepted are based on the ISO-8601 duration format {@code PnDTnHnMn.nS}
+# with days considered to be exactly 24 hours.
+# <p>
+# Examples:
+# <pre>
+# "PT20.345S" -- parses as "20.345 seconds"
+# "PT15M" -- parses as "15 minutes" (where a minute is 60 seconds)
+# "PT10H" -- parses as "10 hours" (where an hour is 3600 seconds)
+# "P2D" -- parses as "2 days" (where a day is 24 hours or 86400
seconds)
+# "P2DT3H4M" -- parses as "2 days, 3 hours and 4 minutes"
+# "P-6H3M" -- parses as "-6 hours and +3 minutes"
+# "-P6H3M" -- parses as "-6 hours and -3 minutes"
+# "-P-6H+3M" -- parses as "+6 hours and -3 minutes"
+# </pre>
+filter: "{ tags -> tags.job_name == 'pulsar-monitoring' }" # The OpenTelemetry
job name
+expSuffix: tag({tags -> tags.cluster = 'pulsar::' +
tags.cluster}).instance(['cluster'], ['node'], Layer.PULSAR)
+metricPrefix: meter_pulsar_broker
+
+# Metrics Rules
+metricsRules:
+
+ # connection Metrics
+ - name: active_connections
+ exp: pulsar_active_connections.sum(['cluster', 'node'])
+ - name: total_connections
+ exp: pulsar_connection_created_total_count.sum(['cluster', 'node'])
+ - name: connection_create_success_count
+ exp: pulsar_connection_create_success_count.sum(['cluster', 'node'])
+ - name: connection_create_fail_count
+ exp: pulsar_connection_create_fail_count.sum(['cluster', 'node'])
+ - name: connection_closed_total_count
+ exp: pulsar_connection_closed_total_count.sum(['cluster', 'node'])
+
+ # JVM Metrics
+ - name: jvm_memory_used
+ exp: jvm_memory_bytes_used.sum(['cluster', 'node'])
+ - name: jvm_memory_committed
+ exp: jvm_memory_bytes_committed.sum(['cluster', 'node'])
+ - name: jvm_memory_init
+ exp: jvm_memory_bytes_init.sum(['cluster', 'node'])
+
+ - name: jvm_memory_pool_used
+ exp: jvm_memory_pool_bytes_used.sum(['cluster', 'node', 'pool'])
+
+ - name: jvm_buffer_pool_used_bytes
+ exp: jvm_buffer_pool_used_bytes.sum(['cluster', 'node', 'pool'])
+
+ - name: jvm_gc_collection_seconds_count
+ exp: jvm_gc_collection_seconds_count.sum(['cluster', 'node',
'gc']).rate('PT1M')
+ - name: jvm_gc_collection_seconds_sum
+ exp: jvm_gc_collection_seconds_sum.sum(['cluster', 'node',
'gc']).rate('PT1M')
+
+ - name: jvm_threads_current
+ exp: jvm_threads_current.sum(['cluster', 'node'])
+ - name: jvm_threads_daemon
+ exp: jvm_threads_daemon.sum(['cluster', 'node'])
+ - name: jvm_threads_peak
+ exp: jvm_threads_peak.sum(['cluster', 'node'])
+ - name: jvm_threads_deadlocked
+ exp: jvm_threads_deadlocked.sum(['cluster', 'node'])
diff --git
a/oap-server/server-starter/src/main/resources/otel-rules/pulsar/pulsar-cluster.yaml
b/oap-server/server-starter/src/main/resources/otel-rules/pulsar/pulsar-cluster.yaml
new file mode 100644
index 0000000000..d5a31c45c1
--- /dev/null
+++
b/oap-server/server-starter/src/main/resources/otel-rules/pulsar/pulsar-cluster.yaml
@@ -0,0 +1,68 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This will parse a textual representation of a duration. The formats
+# accepted are based on the ISO-8601 duration format {@code PnDTnHnMn.nS}
+# with days considered to be exactly 24 hours.
+# <p>
+# Examples:
+# <pre>
+# "PT20.345S" -- parses as "20.345 seconds"
+# "PT15M" -- parses as "15 minutes" (where a minute is 60 seconds)
+# "PT10H" -- parses as "10 hours" (where an hour is 3600 seconds)
+# "P2D" -- parses as "2 days" (where a day is 24 hours or 86400
seconds)
+# "P2DT3H4M" -- parses as "2 days, 3 hours and 4 minutes"
+# "P-6H3M" -- parses as "-6 hours and +3 minutes"
+# "-P6H3M" -- parses as "-6 hours and -3 minutes"
+# "-P-6H+3M" -- parses as "+6 hours and -3 minutes"
+# </pre>
+
+filter: "{ tags -> tags.job_name == 'pulsar-monitoring' }" # The OpenTelemetry
job name
+expSuffix: tag({tags -> tags.cluster = 'pulsar::' +
tags.cluster}).service(['cluster'], Layer.PULSAR)
+metricPrefix: meter_pulsar
+
+# Metrics Rules
+metricsRules:
+
+ # Topic and Subscription Metrics
+ - name: total_topics
+ exp: pulsar_broker_topics_count.sum(['cluster', 'node'])
+ - name: total_subscriptions
+ exp: pulsar_broker_subscriptions_count.sum(['cluster', 'node'])
+
+ # Producer and Consumer Metrics
+ - name: total_producers
+ exp: pulsar_broker_producers_count.sum(['cluster', 'node'])
+ - name: total_consumers
+ exp: pulsar_broker_consumers_count.sum(['cluster', 'node'])
+
+ # Message Rate and Throughput Metrics
+ - name: message_rate_in
+ exp: pulsar_broker_rate_in.sum(['cluster', 'node'])
+ - name: message_rate_out
+ exp: pulsar_broker_rate_out.sum(['cluster', 'node'])
+ - name: throughput_in
+ exp: pulsar_broker_throughput_in.sum(['cluster', 'node'])
+ - name: throughput_out
+ exp: pulsar_broker_throughput_out.sum(['cluster', 'node'])
+
+ - name: storage_size
+ exp: pulsar_broker_storage_size.sum(['cluster', 'node'])
+ - name: storage_logical_size
+ exp: pulsar_broker_storage_logical_size.sum(['cluster', 'node'])
+ - name: storage_write_rate
+ exp: pulsar_broker_storage_write_rate.sum(['cluster', 'node'])
+ - name: storage_read_rate
+ exp: pulsar_broker_storage_read_rate.sum(['cluster', 'node'])
\ No newline at end of file
diff --git
a/oap-server/server-starter/src/main/resources/ui-initialized-templates/bookkeeper/bookkeeper-cluster.json
b/oap-server/server-starter/src/main/resources/ui-initialized-templates/bookkeeper/bookkeeper-cluster.json
new file mode 100644
index 0000000000..ab2e682be2
--- /dev/null
+++
b/oap-server/server-starter/src/main/resources/ui-initialized-templates/bookkeeper/bookkeeper-cluster.json
@@ -0,0 +1,393 @@
+[
+ {
+ "id": "BookKeeper-Cluster",
+ "configuration": {
+ "children": [
+ {
+ "x": 0,
+ "y": 0,
+ "w": 24,
+ "h": 54,
+ "i": "16",
+ "type": "Tab",
+ "children": [
+ {
+ "name": "Overview",
+ "children": [
+ {
+ "x": 0,
+ "y": 0,
+ "w": 12,
+ "h": 11,
+ "i": "7",
+ "type": "Widget",
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "title": "Bookie Ledgers Count",
+ "tips": "",
+ "name": "bookie_ledgers_count"
+ },
+ "expressions": [
+ "meter_bookkeeper_bookie_ledgers_count"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES"
+ ],
+ "metricMode": "Expression",
+ "id": "16-0-7",
+ "moved": false,
+ "metrics": [],
+ "metricTypes": [],
+ "metricConfig": []
+ },
+ {
+ "x": 0,
+ "y": 11,
+ "w": 12,
+ "h": 11,
+ "i": "8",
+ "type": "Widget",
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "title": "Bookie Ledger Writable Dirs",
+ "tips": "",
+ "name": "bookie_ledger_writable_dirs"
+ },
+ "expressions": [
+ "meter_bookkeeper_bookie_ledger_writable_dirs"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES"
+ ],
+ "metricMode": "Expression",
+ "id": "16-0-8",
+ "moved": false,
+ "metricConfig": [],
+ "metricTypes": [],
+ "metrics": []
+ },
+ {
+ "x": 12,
+ "y": 11,
+ "w": 12,
+ "h": 11,
+ "i": "9",
+ "type": "Widget",
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "title": "Bookie Ledger Dir Usage",
+ "tips": "",
+ "name": "bookie_ledger_dir_data_bookkeeper_ledgers_usage"
+ },
+ "expressions": [
+
"meter_bookkeeper_bookie_ledger_dir_data_bookkeeper_ledgers_usage"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES"
+ ],
+ "metricMode": "Expression",
+ "id": "16-0-9",
+ "moved": false,
+ "metricConfig": [],
+ "metricTypes": [],
+ "metrics": []
+ },
+ {
+ "x": 12,
+ "y": 0,
+ "w": 12,
+ "h": 11,
+ "i": "10",
+ "type": "Widget",
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "title": "Bookie Entries Count",
+ "tips": "",
+ "name": "bookie_entries_count"
+ },
+ "expressions": [
+ "meter_bookkeeper_bookie_entries_count"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES"
+ ],
+ "metricMode": "Expression",
+ "id": "16-0-10",
+ "moved": false,
+ "metricConfig": [],
+ "metricTypes": [],
+ "metrics": []
+ },
+ {
+ "x": 0,
+ "y": 22,
+ "w": 6,
+ "h": 11,
+ "i": "11",
+ "type": "Widget",
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "title": "Bookie Write Cache Size (MB)",
+ "tips": "",
+ "name": "bookie_write_cache_size"
+ },
+ "expressions": [
+ "meter_bookkeeper_bookie_write_cache_size/1024/1024"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES"
+ ],
+ "metricMode": "Expression",
+ "id": "16-0-11",
+ "moved": false,
+ "metricConfig": [
+ {
+ "label": "read cache size"
+ }
+ ],
+ "metricTypes": [],
+ "metrics": []
+ },
+ {
+ "x": 12,
+ "y": 22,
+ "w": 6,
+ "h": 11,
+ "i": "12",
+ "type": "Widget",
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "title": "Bookie Write Cache Entry Count",
+ "tips": "",
+ "name": "bookie_write_cache_count"
+ },
+ "expressions": [
+ "meter_bookkeeper_bookie_write_cache_count"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES"
+ ],
+ "metricMode": "Expression",
+ "id": "16-0-12",
+ "moved": false,
+ "metricConfig": [
+ {
+ "label": "read cache count"
+ }
+ ],
+ "metricTypes": [],
+ "metrics": []
+ },
+ {
+ "x": 6,
+ "y": 22,
+ "w": 6,
+ "h": 11,
+ "i": "13",
+ "type": "Widget",
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "title": "Bookie Read Cache (MB)",
+ "tips": "",
+ "name": "_bookie_read_cache_size"
+ },
+ "expressions": [
+ "meter_bookkeeper_bookie_read_cache_size/1024/1024"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES"
+ ],
+ "metricMode": "Expression",
+ "id": "16-0-13",
+ "moved": false,
+ "metricConfig": [],
+ "metricTypes": [],
+ "metrics": []
+ },
+ {
+ "x": 18,
+ "y": 22,
+ "w": 6,
+ "h": 11,
+ "i": "14",
+ "type": "Widget",
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "title": "Bookie Read Cache Entry Count",
+ "tips": "",
+ "name": "throughput_out"
+ },
+ "expressions": [
+ "meter_bookkeeper_bookie_read_cache_count"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES"
+ ],
+ "metricMode": "Expression",
+ "id": "16-0-14",
+ "moved": false,
+ "metricConfig": [],
+ "metricTypes": [],
+ "metrics": []
+ },
+ {
+ "x": 12,
+ "y": 33,
+ "w": 12,
+ "h": 10,
+ "i": "16",
+ "type": "Widget",
+ "id": "16-0-16",
+ "metricTypes": [],
+ "metrics": [],
+ "metricMode": "Expression",
+ "moved": false,
+ "expressions": [
+ "meter_bookkeeper_bookie_read_rate"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES"
+ ],
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "name": "bookie_read_rate",
+ "title": "Bookie Read Rate(bytes/s)"
+ }
+ },
+ {
+ "x": 0,
+ "y": 33,
+ "w": 12,
+ "h": 10,
+ "i": "18",
+ "type": "Widget",
+ "id": "16-0-18",
+ "metricTypes": [],
+ "metrics": [],
+ "metricMode": "Expression",
+ "moved": false,
+ "expressions": [
+ "meter_bookkeeper_bookie_write_rate"
+ ],
+ "metricConfig": [],
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true,
+ "legend": {
+ "asTable": false
+ }
+ },
+ "widget": {
+ "name": "bookie_write_rate",
+ "title": "Bookie Write Rate(bytes/s)"
+ },
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES"
+ ]
+ }
+ ]
+ },
+ {
+ "name": "Node",
+ "children": [
+ {
+ "x": 0,
+ "y": 0,
+ "w": 24,
+ "h": 48,
+ "i": "0",
+ "type": "Widget",
+ "graph": {
+ "type": "InstanceList",
+ "dashboardName": "BookKeeper-Node",
+ "fontSize": 12
+ },
+ "metrics": [],
+ "metricTypes": [],
+ "metricConfig": [],
+ "id": "16-1-0",
+ "moved": false
+ }
+ ]
+ }
+ ],
+ "id": "16",
+ "activedTabIndex": 0,
+ "moved": false
+ }
+ ],
+ "layer": "BOOKKEEPER",
+ "entity": "Service",
+ "name": "BookKeeper-Cluster",
+ "id": "BookKeeper-Cluster",
+ "isRoot": false
+ }
+ }
+]
\ No newline at end of file
diff --git
a/oap-server/server-starter/src/main/resources/ui-initialized-templates/bookkeeper/bookkeeper-node.json
b/oap-server/server-starter/src/main/resources/ui-initialized-templates/bookkeeper/bookkeeper-node.json
new file mode 100644
index 0000000000..2d76bfad7a
--- /dev/null
+++
b/oap-server/server-starter/src/main/resources/ui-initialized-templates/bookkeeper/bookkeeper-node.json
@@ -0,0 +1,366 @@
+[
+ {
+ "id": "BookKeeper-Node",
+ "configuration": {
+ "children": [
+ {
+ "x": 12,
+ "y": 22,
+ "w": 12,
+ "h": 11,
+ "i": "28",
+ "type": "Widget",
+ "id": "28",
+ "metricTypes": [],
+ "metrics": [],
+ "metricMode": "Expression",
+ "moved": false,
+ "expressions": [
+ "meter_bookkeeper_node_jvm_memory_pool_used/1024/1024"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES"
+ ],
+ "graph": {
+ "type": "Area",
+ "opacity": 0.4,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "name": "jvm_memory_pool_used",
+ "title": "JVM Memory Pool Used (MB)"
+ }
+ },
+ {
+ "x": 0,
+ "y": 22,
+ "w": 12,
+ "h": 11,
+ "i": "29",
+ "type": "Widget",
+ "id": "29",
+ "metricTypes": [],
+ "metrics": [],
+ "metricMode": "Expression",
+ "moved": false,
+ "expressions": [
+ "meter_bookkeeper_node_jvm_memory_used/1024/1024",
+ "meter_bookkeeper_node_jvm_memory_committed/1024/1024",
+ "meter_bookkeeper_node_jvm_memory_init/1024/1024"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES",
+ "TIME_SERIES_VALUES",
+ "TIME_SERIES_VALUES"
+ ],
+ "graph": {
+ "type": "Area",
+ "opacity": 0.4,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "metricConfig": [
+ {
+ "label": "used",
+ "unit": "MB"
+ },
+ {
+ "label": "committed",
+ "unit": "MB"
+ },
+ {
+ "unit": "MB",
+ "label": "init"
+ }
+ ],
+ "widget": {
+ "name": "jvm_memory",
+ "title": "JVM Memory (MB)"
+ }
+ },
+ {
+ "x": 16,
+ "y": 33,
+ "w": 8,
+ "h": 11,
+ "i": "30",
+ "type": "Widget",
+ "id": "30",
+ "metricTypes": [],
+ "metrics": [],
+ "metricMode": "Expression",
+ "moved": false,
+ "expressions": [
+ "meter_bookkeeper_node_jvm_threads_current",
+ "meter_bookkeeper_node_jvm_threads_daemon",
+ "meter_bookkeeper_node_jvm_threads_peak",
+ "meter_bookkeeper_node_jvm_threads_deadlocked"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES",
+ "TIME_SERIES_VALUES",
+ "TIME_SERIES_VALUES",
+ "TIME_SERIES_VALUES"
+ ],
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "metricConfig": [
+ {
+ "label": "current threads"
+ },
+ {
+ "label": "daemon threads"
+ },
+ {
+ "label": "peak threads"
+ },
+ {
+ "label": "deadlocked threads"
+ }
+ ],
+ "widget": {
+ "name": "meter_bookkeeper_node_jvm_threads",
+ "title": "JVM Threads"
+ }
+ },
+ {
+ "x": 8,
+ "y": 33,
+ "w": 8,
+ "h": 11,
+ "i": "31",
+ "type": "Widget",
+ "id": "31",
+ "metricTypes": [],
+ "metrics": [],
+ "metricMode": "Expression",
+ "moved": false,
+ "expressions": [
+ "meter_bookkeeper_node_jvm_gc_collection_seconds_sum"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES"
+ ],
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "name": "jvm_gc_collection_seconds_sum",
+ "title": "GC Time (s)"
+ }
+ },
+ {
+ "x": 0,
+ "y": 33,
+ "w": 8,
+ "h": 11,
+ "i": "32",
+ "type": "Widget",
+ "id": "32",
+ "metricTypes": [],
+ "metrics": [],
+ "metricMode": "Expression",
+ "moved": false,
+ "expressions": [
+ "meter_bookkeeper_node_jvm_gc_collection_seconds_count"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES"
+ ],
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "name": "jvm_gc_collection_seconds_count",
+ "title": "GC Count"
+ }
+ },
+ {
+ "x": 0,
+ "y": 0,
+ "w": 12,
+ "h": 11,
+ "i": "33",
+ "type": "Widget",
+ "id": "33",
+ "metricTypes": [],
+ "metrics": [],
+ "metricMode": "Expression",
+ "moved": false,
+ "expressions": [
+
"meter_bookkeeper_node_thread_executor_completed.sum(['cluster','node'])"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES"
+ ],
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "metricConfig": [
+ {
+ "label": "completed"
+ }
+ ],
+ "widget": {
+ "name": "thread_executor_completed",
+ "title": "Thread Executor Completed"
+ }
+ },
+ {
+ "x": 12,
+ "y": 0,
+ "w": 12,
+ "h": 11,
+ "i": "34",
+ "type": "Widget",
+ "id": "34",
+ "metricTypes": [],
+ "metrics": [],
+ "metricMode": "Expression",
+ "moved": false,
+ "expressions": [
+ "meter_bookkeeper_node_thread_executor_tasks_completed",
+ "meter_bookkeeper_node_thread_executor_tasks_rejected",
+ "meter_bookkeeper_node_thread_executor_tasks_failed"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES",
+ "TIME_SERIES_VALUES",
+ "TIME_SERIES_VALUES"
+ ],
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "metricConfig": [
+ {
+ "label": "completed"
+ },
+ {
+ "label": "rejected"
+ },
+ {
+ "label": "failed"
+ }
+ ],
+ "widget": {
+ "name": "thread_executor_tasks",
+ "title": "Thread Executor Tasks"
+ }
+ },
+ {
+ "x": 0,
+ "y": 11,
+ "w": 12,
+ "h": 11,
+ "i": "35",
+ "type": "Widget",
+ "id": "35",
+ "metricTypes": [],
+ "metrics": [],
+ "metricMode": "Expression",
+ "moved": false,
+ "expressions": [
+ "meter_bookkeeper_node_high_priority_threads",
+ "meter_bookkeeper_node_read_thread_pool_threads"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES",
+ "TIME_SERIES_VALUES"
+ ],
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "metricConfig": [
+ {
+ "label": "high priority threads"
+ },
+ {
+ "label": "read thread pool threads"
+ }
+ ],
+ "widget": {
+ "name": "pooled_threads",
+ "title": "Pooled Threads"
+ }
+ },
+ {
+ "x": 12,
+ "y": 11,
+ "w": 12,
+ "h": 11,
+ "i": "36",
+ "type": "Widget",
+ "id": "36",
+ "metricTypes": [],
+ "metrics": [],
+ "metricMode": "Expression",
+ "moved": false,
+ "expressions": [
+ "meter_bookkeeper_node_high_priority_thread_max_queue_size",
+ "meter_bookkeeper_node_read_thread_pool_max_queue_size"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES",
+ "TIME_SERIES_VALUES"
+ ],
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "metricConfig": [
+ {
+ "label": "high_priority_thread max queue size"
+ },
+ {
+ "label": "read_thread_pool max queue size"
+ }
+ ],
+ "widget": {
+ "name": "max_queue_size",
+ "title": "Pooled Threads Max Queue Size"
+ }
+ }
+ ],
+ "layer": "BOOKKEEPER",
+ "entity": "ServiceInstance",
+ "name": "BookKeeper-Node",
+ "id": "BookKeeper-Node"
+ }
+ }
+]
diff --git
a/oap-server/server-starter/src/main/resources/ui-initialized-templates/bookkeeper/bookkeeper-root.json
b/oap-server/server-starter/src/main/resources/ui-initialized-templates/bookkeeper/bookkeeper-root.json
new file mode 100644
index 0000000000..93caeda455
--- /dev/null
+++
b/oap-server/server-starter/src/main/resources/ui-initialized-templates/bookkeeper/bookkeeper-root.json
@@ -0,0 +1,51 @@
+[
+ {
+ "id": "BookKeeper-Root",
+ "configuration": {
+ "children": [
+ {
+ "x": 0,
+ "y": 3,
+ "w": 24,
+ "h": 29,
+ "i": "0",
+ "type": "Widget",
+ "id": "0",
+ "metricTypes": [
+ ""
+ ],
+ "metrics": [
+ ""
+ ],
+ "graph": {
+ "type": "ServiceList",
+ "dashboardName": "BookKeeper-Cluster",
+ "fontSize": 12,
+ "showXAxis": false,
+ "showYAxis": false,
+ "showGroup": true
+ }
+ }, {
+ "x": 0,
+ "y": 0,
+ "w": 24,
+ "h": 3,
+ "i": "1",
+ "type": "Text",
+ "graph": {
+ "fontColor": "blue",
+ "backgroundColor": "white",
+ "content": "Provide BookKeeper monitoring through OpenTelemetry's
Prometheus Receiver",
+ "fontSize": 14,
+ "textAlign": "left",
+ "url":
"https://skywalking.apache.org/docs/main/next/en/setup/backend/backend-bookkeeper-monitoring/";
+ }
+ }
+ ],
+ "layer": "BOOKKEEPER",
+ "entity": "All",
+ "name": "BookKeeper-Root",
+ "isRoot": true
+ }
+ }
+]
diff --git
a/oap-server/server-starter/src/main/resources/ui-initialized-templates/menu.yaml
b/oap-server/server-starter/src/main/resources/ui-initialized-templates/menu.yaml
index 673df5d122..3ae556d889 100644
---
a/oap-server/server-starter/src/main/resources/ui-initialized-templates/menu.yaml
+++
b/oap-server/server-starter/src/main/resources/ui-initialized-templates/menu.yaml
@@ -188,6 +188,11 @@ menus:
description: Provide MongoDB monitoring through OpenTelemetry's
Prometheus Receiver.
documentLink:
https://skywalking.apache.org/docs/main/next/en/setup/backend/backend-mongodb-monitoring/
i18nKey: database_mongodb
+ - title: BookKeeper
+ layer: BOOKKEEPER
+ description: Provide BookKeeper monitoring through OpenTelemetry's
Prometheus Receiver.
+ documentLink:
https://skywalking.apache.org/docs/main/next/en/setup/backend/backend-bookkeeper-monitoring/
+ i18nKey: database_bookkeeper
- title: Message Queue
icon: mq
description: A message queue is a form of asynchronous service-to-service
communication used in serverless and microservices architectures.
@@ -203,6 +208,11 @@ menus:
description: Provide Kafka monitoring through OpenTelemetry's
Prometheus Receiver.
documentLink:
https://skywalking.apache.org/docs/main/next/en/setup/backend/backend-kafka-monitoring/
i18nKey: mq_kafka
+ - title: Pulsar
+ layer: PULSAR
+ description: Provide pulsar monitoring through OpenTelemetry's
Prometheus Receiver.
+ documentLink:
https://skywalking.apache.org/docs/main/next/en/setup/backend/backend-pulsar-monitoring/
+ i18nKey: mq_pulsar
- title: Self Observability
icon: self_observability
description: Self Observability provides the observabilities for running
components and servers from the SkyWalking ecosystem.
diff --git
a/oap-server/server-starter/src/main/resources/ui-initialized-templates/pulsar/pulsar-broker.json
b/oap-server/server-starter/src/main/resources/ui-initialized-templates/pulsar/pulsar-broker.json
new file mode 100644
index 0000000000..4f6c65ceff
--- /dev/null
+++
b/oap-server/server-starter/src/main/resources/ui-initialized-templates/pulsar/pulsar-broker.json
@@ -0,0 +1,422 @@
+[
+ {
+ "id": "Pulsar-Broker",
+ "configuration": {
+ "children": [
+ {
+ "x": 0,
+ "y": 0,
+ "w": 12,
+ "h": 11,
+ "i": "9",
+ "type": "Widget",
+ "widget": {
+ "title": "Active Connections",
+ "tips": "Number of active connections",
+ "name": "active_connections"
+ },
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "expressions": [
+ "meter_pulsar_broker_active_connections"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES"
+ ],
+ "metricMode": "Expression",
+ "id": "9",
+ "moved": false,
+ "associate": [],
+ "metricConfig": [
+ {
+ "label": "active_connections"
+ }
+ ],
+ "metrics": [],
+ "metricTypes": []
+ },
+ {
+ "x": 12,
+ "y": 0,
+ "w": 12,
+ "h": 11,
+ "i": "10",
+ "type": "Widget",
+ "widget": {
+ "title": "Total Connections",
+ "tips": "Total number of connections",
+ "name": "total_connections"
+ },
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "expressions": [
+ "meter_pulsar_broker_total_connections"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES"
+ ],
+ "metricMode": "Expression",
+ "id": "10",
+ "moved": false,
+ "metricConfig": [
+ {
+ "label": "total_connections"
+ }
+ ],
+ "metrics": [],
+ "metricTypes": []
+ },
+ {
+ "x": 8,
+ "y": 11,
+ "w": 8,
+ "h": 11,
+ "i": "22",
+ "type": "Widget",
+ "widget": {
+ "title": "Connection Create Fail Count",
+ "tips": "",
+ "name": "connection_create_fail_count"
+ },
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "expressions": [
+
"meter_pulsar_broker_connection_create_fail_count.sum(['cluster','broker'])"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES"
+ ],
+ "metricMode": "Expression",
+ "id": "22",
+ "moved": false,
+ "metrics": [],
+ "metricTypes": [],
+ "metricConfig": [
+ {
+ "label": "connection_create_fail"
+ }
+ ]
+ },
+ {
+ "x": 0,
+ "y": 11,
+ "w": 8,
+ "h": 11,
+ "i": "25",
+ "type": "Widget",
+ "widget": {
+ "title": "Connection Create Success Count",
+ "tips": "Peak JVM threads",
+ "name": "connection_create_success_count"
+ },
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true,
+ "legend": {
+ "asTable": false,
+ "show": true
+ }
+ },
+ "expressions": [
+ "meter_pulsar_broker_connection_create_success_count"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES"
+ ],
+ "metricMode": "Expression",
+ "id": "25",
+ "moved": false,
+ "metricTypes": [],
+ "metrics": [],
+ "associate": [],
+ "metricConfig": [
+ {
+ "label": "connection_create_success"
+ }
+ ]
+ },
+ {
+ "x": 16,
+ "y": 11,
+ "w": 8,
+ "h": 11,
+ "i": "26",
+ "type": "Widget",
+ "widget": {
+ "title": "Connection Closed Total Count",
+ "tips": "connection_closed_total_count",
+ "name": "connection_closed_total_count"
+ },
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "expressions": [
+ "meter_pulsar_broker_connection_closed_total_count"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES"
+ ],
+ "metricMode": "Expression",
+ "id": "26",
+ "moved": false,
+ "metrics": [],
+ "metricTypes": [],
+ "metricConfig": [
+ {
+ "label": "connection_closed"
+ }
+ ]
+ },
+ {
+ "x": 16,
+ "y": 22,
+ "w": 8,
+ "h": 11,
+ "i": "27",
+ "type": "Widget",
+ "id": "27",
+ "metricTypes": [],
+ "metrics": [],
+ "metricMode": "Expression",
+ "moved": false,
+ "expressions": [
+ "meter_pulsar_broker_jvm_buffer_pool_used_bytes/1024/1024"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES"
+ ],
+ "graph": {
+ "type": "Area",
+ "opacity": 0.4,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "name": "jvm_buffer_pool_used_bytes",
+ "title": "JVM Buffer Pool Used (MB)"
+ }
+ },
+ {
+ "x": 8,
+ "y": 22,
+ "w": 8,
+ "h": 11,
+ "i": "28",
+ "type": "Widget",
+ "id": "28",
+ "metricTypes": [],
+ "metrics": [],
+ "metricMode": "Expression",
+ "moved": false,
+ "expressions": [
+ "meter_pulsar_broker_jvm_memory_pool_used/1024/1024"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES"
+ ],
+ "graph": {
+ "type": "Area",
+ "opacity": 0.4,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "name": "jvm_memory_pool_used",
+ "title": "JVM Memory Pool Used (MB)"
+ }
+ },
+ {
+ "x": 0,
+ "y": 22,
+ "w": 8,
+ "h": 11,
+ "i": "29",
+ "type": "Widget",
+ "id": "29",
+ "metricTypes": [],
+ "metrics": [],
+ "metricMode": "Expression",
+ "moved": false,
+ "expressions": [
+ "meter_pulsar_broker_jvm_memory_used/1024/1024",
+ "meter_pulsar_broker_jvm_memory_committed/1024/1024",
+ "meter_pulsar_broker_jvm_memory_init/1024/1024"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES",
+ "TIME_SERIES_VALUES",
+ "TIME_SERIES_VALUES"
+ ],
+ "graph": {
+ "type": "Area",
+ "opacity": 0.4,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "metricConfig": [
+ {
+ "label": "used",
+ "unit": "MB"
+ },
+ {
+ "label": "committed",
+ "unit": "MB"
+ },
+ {
+ "unit": "MB",
+ "label": "init"
+ }
+ ],
+ "widget": {
+ "name": "jvm_memory",
+ "title": "JVM Memory (MB)"
+ }
+ },
+ {
+ "x": 16,
+ "y": 33,
+ "w": 8,
+ "h": 11,
+ "i": "30",
+ "type": "Widget",
+ "id": "30",
+ "metricTypes": [],
+ "metrics": [],
+ "metricMode": "Expression",
+ "moved": false,
+ "expressions": [
+ "meter_pulsar_broker_jvm_threads_current",
+ "meter_pulsar_broker_jvm_threads_daemon",
+ "meter_pulsar_broker_jvm_threads_peak",
+ "meter_pulsar_broker_jvm_threads_deadlocked"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES",
+ "TIME_SERIES_VALUES",
+ "TIME_SERIES_VALUES",
+ "TIME_SERIES_VALUES"
+ ],
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "metricConfig": [
+ {
+ "label": "current threads"
+ },
+ {
+ "label": "daemon threads"
+ },
+ {
+ "label": "peak threads"
+ },
+ {
+ "label": "deadlocked threads"
+ }
+ ],
+ "widget": {
+ "name": "meter_pulsar_broker_jvm_threads",
+ "title": "JVM Threads"
+ }
+ },
+ {
+ "x": 8,
+ "y": 33,
+ "w": 8,
+ "h": 11,
+ "i": "31",
+ "type": "Widget",
+ "id": "31",
+ "metricTypes": [],
+ "metrics": [],
+ "metricMode": "Expression",
+ "moved": false,
+ "expressions": [
+ "meter_pulsar_broker_jvm_gc_collection_seconds_sum"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES"
+ ],
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "name": "jvm_gc_collection_seconds_sum",
+ "title": "GC Time (ms/min)"
+ }
+ },
+ {
+ "x": 0,
+ "y": 33,
+ "w": 8,
+ "h": 11,
+ "i": "32",
+ "type": "Widget",
+ "id": "32",
+ "metricTypes": [],
+ "metrics": [],
+ "metricMode": "Expression",
+ "moved": false,
+ "expressions": [
+ "meter_pulsar_broker_jvm_gc_collection_seconds_count"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES"
+ ],
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "name": "jvm_gc_collection_seconds_count",
+ "title": "GC Count"
+ }
+ }
+ ],
+ "layer": "PULSAR",
+ "entity": "ServiceInstance",
+ "name": "Pulsar-Broker",
+ "id": "Pulsar-Broker"
+ }
+ }
+]
diff --git
a/oap-server/server-starter/src/main/resources/ui-initialized-templates/pulsar/pulsar-cluster.json
b/oap-server/server-starter/src/main/resources/ui-initialized-templates/pulsar/pulsar-cluster.json
new file mode 100644
index 0000000000..71003854b7
--- /dev/null
+++
b/oap-server/server-starter/src/main/resources/ui-initialized-templates/pulsar/pulsar-cluster.json
@@ -0,0 +1,435 @@
+[
+ {
+ "id": "Pulsar-Cluster",
+ "configuration": {
+ "children": [
+ {
+ "x": 0,
+ "y": 0,
+ "w": 24,
+ "h": 54,
+ "i": "16",
+ "type": "Tab",
+ "children": [
+ {
+ "name": "Overview",
+ "children": [
+ {
+ "x": 0,
+ "y": 0,
+ "w": 6,
+ "h": 10,
+ "i": "7",
+ "type": "Widget",
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "title": "Total Topics",
+ "tips": "Total number of topics.",
+ "name": "total_topics"
+ },
+ "expressions": [
+ "meter_pulsar_total_topics"
+ ],
+ "typesOfMQE": [],
+ "metricMode": "Expression",
+ "id": "16-0-7",
+ "moved": false,
+ "metrics": [],
+ "metricTypes": [],
+ "metricConfig": []
+ },
+ {
+ "x": 6,
+ "y": 0,
+ "w": 6,
+ "h": 10,
+ "i": "8",
+ "type": "Widget",
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "title": "Total Subscriptions",
+ "tips": "Total number of subscriptions.",
+ "name": "total_subscriptions"
+ },
+ "expressions": [
+ "meter_pulsar_total_subscriptions"
+ ],
+ "typesOfMQE": [],
+ "metricMode": "Expression",
+ "id": "16-0-8",
+ "moved": false,
+ "metricConfig": [],
+ "metricTypes": [],
+ "metrics": []
+ },
+ {
+ "x": 12,
+ "y": 0,
+ "w": 6,
+ "h": 10,
+ "i": "9",
+ "type": "Widget",
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "title": "Total Producers",
+ "tips": "Total number of producers.",
+ "name": "total_producers"
+ },
+ "expressions": [
+ "meter_pulsar_total_producers"
+ ],
+ "typesOfMQE": [],
+ "metricMode": "Expression",
+ "id": "16-0-9",
+ "moved": false,
+ "metricConfig": [],
+ "metricTypes": [],
+ "metrics": []
+ },
+ {
+ "x": 18,
+ "y": 0,
+ "w": 6,
+ "h": 10,
+ "i": "10",
+ "type": "Widget",
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "title": "Total Consumers",
+ "tips": "Total number of consumers.",
+ "name": "total_consumers"
+ },
+ "expressions": [
+ "meter_pulsar_total_consumers"
+ ],
+ "typesOfMQE": [],
+ "metricMode": "Expression",
+ "id": "16-0-10",
+ "moved": false,
+ "metricConfig": [],
+ "metricTypes": [],
+ "metrics": []
+ },
+ {
+ "x": 0,
+ "y": 10,
+ "w": 12,
+ "h": 10,
+ "i": "11",
+ "type": "Widget",
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "title": "Message Rate In",
+ "tips": "Incoming message rate.",
+ "name": "message_rate_in"
+ },
+ "expressions": [
+ "meter_pulsar_message_rate_in"
+ ],
+ "typesOfMQE": [],
+ "metricMode": "Expression",
+ "id": "16-0-11",
+ "moved": false,
+ "metricConfig": [],
+ "metricTypes": [
+ "readLabeledMetricsValues"
+ ],
+ "metrics": [
+ "meter_pulsar_message_rate_in"
+ ]
+ },
+ {
+ "x": 12,
+ "y": 10,
+ "w": 12,
+ "h": 10,
+ "i": "12",
+ "type": "Widget",
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "title": "Message Rate Out",
+ "tips": "Outgoing message rate.",
+ "name": "message_rate_out"
+ },
+ "expressions": [
+ "meter_pulsar_message_rate_in"
+ ],
+ "typesOfMQE": [],
+ "metricMode": "Expression",
+ "id": "16-0-12",
+ "moved": false,
+ "metricConfig": [],
+ "metricTypes": [],
+ "metrics": []
+ },
+ {
+ "x": 0,
+ "y": 20,
+ "w": 12,
+ "h": 10,
+ "i": "13",
+ "type": "Widget",
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "title": "Throughput In",
+ "tips": "Incoming throughput.",
+ "name": "throughput_in"
+ },
+ "expressions": [
+ "meter_pulsar_throughput_in"
+ ],
+ "typesOfMQE": [],
+ "metricMode": "Expression",
+ "id": "16-0-13",
+ "moved": false,
+ "metricConfig": [],
+ "metricTypes": [],
+ "metrics": []
+ },
+ {
+ "x": 12,
+ "y": 20,
+ "w": 12,
+ "h": 10,
+ "i": "14",
+ "type": "Widget",
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "title": "Throughput Out",
+ "tips": "Outgoing throughput.",
+ "name": "throughput_out"
+ },
+ "expressions": [
+ "meter_pulsar_throughput_in"
+ ],
+ "typesOfMQE": [],
+ "metricMode": "Expression",
+ "id": "16-0-14",
+ "moved": false,
+ "metricConfig": [],
+ "metricTypes": [],
+ "metrics": []
+ },
+ {
+ "x": 12,
+ "y": 40,
+ "w": 12,
+ "h": 10,
+ "i": "15",
+ "type": "Widget",
+ "id": "16-0-15",
+ "metricTypes": [],
+ "metrics": [],
+ "metricMode": "Expression",
+ "moved": false,
+ "expressions": [
+ "meter_pulsar_storage_read_rate.sum(['cluster','node'])"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES"
+ ],
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "name": "storage_read_rate",
+ "title": "Storage Read Rate (messages per second)"
+ }
+ },
+ {
+ "x": 12,
+ "y": 30,
+ "w": 12,
+ "h": 10,
+ "i": "16",
+ "type": "Widget",
+ "id": "16-0-16",
+ "metricTypes": [],
+ "metrics": [],
+ "metricMode": "Expression",
+ "moved": false,
+ "expressions": [
+
"meter_pulsar_storage_logical_size.sum(['cluster','node'])/1024/1024"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES"
+ ],
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "name": "storage_logical_size",
+ "title": "Storage Logical Size (MB)"
+ }
+ },
+ {
+ "x": 0,
+ "y": 40,
+ "w": 12,
+ "h": 10,
+ "i": "17",
+ "type": "Widget",
+ "id": "16-0-17",
+ "metricTypes": [],
+ "metrics": [],
+ "metricMode": "Expression",
+ "moved": false,
+ "expressions": [
+ "meter_pulsar_storage_write_rate.sum(['cluster','node'])"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES"
+ ],
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "name": "storage_write_rate",
+ "title": "Storage Write Rate (messages per second)"
+ }
+ },
+ {
+ "x": 0,
+ "y": 30,
+ "w": 12,
+ "h": 10,
+ "i": "18",
+ "type": "Widget",
+ "id": "16-0-18",
+ "metricTypes": [
+ "readLabeledMetricsValues"
+ ],
+ "metrics": [],
+ "metricMode": "Expression",
+ "moved": false,
+ "expressions": [
+ "meter_pulsar_storage_size/1024/1024"
+ ],
+ "metricConfig": [],
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": true,
+ "showXAxis": true,
+ "showYAxis": true,
+ "legend": {
+ "asTable": false
+ }
+ },
+ "widget": {
+ "name": "storage_size",
+ "title": "Storage Size (MB)"
+ },
+ "typesOfMQE": []
+ }
+ ]
+ },
+ {
+ "name": "Node",
+ "children": [
+ {
+ "x": 0,
+ "y": 0,
+ "w": 24,
+ "h": 48,
+ "i": "0",
+ "type": "Widget",
+ "graph": {
+ "type": "InstanceList",
+ "dashboardName": "Pulsar-Broker",
+ "fontSize": 12
+ },
+ "metrics": [],
+ "metricTypes": [],
+ "metricConfig": [],
+ "id": "16-1-0",
+ "moved": false
+ }
+ ]
+ }
+ ],
+ "id": "16",
+ "activedTabIndex": 0,
+ "moved": false
+ }
+ ],
+ "layer": "PULSAR",
+ "entity": "Service",
+ "name": "Pulsar-Cluster",
+ "id": "Pulsar-Cluster",
+ "isRoot": false
+ }
+ }
+]
\ No newline at end of file
diff --git
a/oap-server/server-starter/src/main/resources/ui-initialized-templates/pulsar/pulsar-root.json
b/oap-server/server-starter/src/main/resources/ui-initialized-templates/pulsar/pulsar-root.json
new file mode 100644
index 0000000000..5126ecb55d
--- /dev/null
+++
b/oap-server/server-starter/src/main/resources/ui-initialized-templates/pulsar/pulsar-root.json
@@ -0,0 +1,51 @@
+[
+ {
+ "id": "Pulsar-Root",
+ "configuration": {
+ "children": [
+ {
+ "x": 0,
+ "y": 3,
+ "w": 24,
+ "h": 29,
+ "i": "0",
+ "type": "Widget",
+ "id": "0",
+ "metricTypes": [
+ ""
+ ],
+ "metrics": [
+ ""
+ ],
+ "graph": {
+ "type": "ServiceList",
+ "dashboardName": "Pulsar-Cluster",
+ "fontSize": 12,
+ "showXAxis": false,
+ "showYAxis": false,
+ "showGroup": true
+ }
+ }, {
+ "x": 0,
+ "y": 0,
+ "w": 24,
+ "h": 3,
+ "i": "1",
+ "type": "Text",
+ "graph": {
+ "fontColor": "blue",
+ "backgroundColor": "white",
+ "content": "Provide Pulsar monitoring through OpenTelemetry's
Prometheus Receiver",
+ "fontSize": 14,
+ "textAlign": "left",
+ "url":
"https://skywalking.apache.org/docs/main/next/en/setup/backend/backend-pulsar-monitoring/";
+ }
+ }
+ ],
+ "layer": "PULSAR",
+ "entity": "All",
+ "name": "Pulsar-Root",
+ "isRoot": true
+ }
+ }
+]
diff --git a/test/e2e-v2/cases/pulsar/docker-compose.yml
b/test/e2e-v2/cases/pulsar/docker-compose.yml
new file mode 100644
index 0000000000..80b870d1dd
--- /dev/null
+++ b/test/e2e-v2/cases/pulsar/docker-compose.yml
@@ -0,0 +1,137 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: '3'
+
+services:
+ oap:
+ extends:
+ file: ../../script/docker-compose/base-compose.yml
+ service: oap
+ ports:
+ - "12800:12800"
+ networks:
+ - e2e
+
+ # Start zookeeper
+ zookeeper:
+ image: zookeeper:3.9.1
+ expose:
+ - 2181
+ networks:
+ - e2e
+ environment:
+ - ALLOW_ANONYMOUS_LOGIN=yes
+ healthcheck:
+ test: ["CMD", "nc", "-nz", "127.0.0.1", "2181"]
+ interval: 15s
+ timeout: 10s
+ retries: 120
+
+ # Start bookie
+ bookie:
+ image: apachepulsar/pulsar:3.1.1
+ restart: on-failure
+ hostname: bookie
+ networks:
+ - e2e
+ environment:
+ - clusterName=pulsar-cluster
+ - zkServers=zookeeper:2181
+ - metadataServiceUri=metadata-store:zk:zookeeper:2181
+ - advertisedAddress=bookie
+ - BOOKIE_MEM=-Xms512m -Xmx512m -XX:MaxDirectMemorySize=256m
+ - allowLoopback=true
+ depends_on:
+ zookeeper:
+ condition: service_healthy
+ expose:
+ - 8000
+ ports:
+ - "8000:8000"
+ command: bash -c "bin/apply-config-from-env.py conf/bookkeeper.conf &&
exec bin/pulsar bookie"
+ healthcheck:
+ test: ["CMD", "nc", "-nz", "127.0.0.1", "8000"]
+ interval: 15s
+ timeout: 10s
+ retries: 120
+
+ # Start broker
+ broker:
+ image: apachepulsar/pulsar:3.1.1
+ restart: on-failure
+ hostname: broker
+ networks:
+ - e2e
+ environment:
+ - metadataStoreUrl=zk:zookeeper:2181
+ - zookeeperServers=zookeeper:2181
+ - clusterName=pulsar-cluster
+ - managedLedgerDefaultEnsembleSize=1
+ - managedLedgerDefaultWriteQuorum=1
+ - managedLedgerDefaultAckQuorum=1
+ - advertisedAddress=broker
+ - advertisedListeners=external:pulsar://127.0.0.1:6650
+ - PULSAR_MEM=-Xms512m -Xmx512m -XX:MaxDirectMemorySize=256m
+ depends_on:
+ zookeeper:
+ condition: service_healthy
+ expose:
+ - 8080
+ - 6650
+ ports:
+ - "6650:6650"
+ - "8080:8080"
+ command: bash -c "bin/pulsar initialize-cluster-metadata --cluster
pulsar-cluster --zookeeper zookeeper:2181 --configuration-store zookeeper:2181
--web-service-url http://broker:8080 --broker-service-url pulsar://broker:6650
&& bin/apply-config-from-env.py conf/broker.conf && exec bin/pulsar broker"
+ healthcheck:
+ test: ["CMD", "nc", "-nz", "127.0.0.1", "8080"]
+ interval: 15s
+ timeout: 10s
+ retries: 120
+
+ consume-perf:
+ image: apachepulsar/pulsar:3.1.1
+ networks:
+ - e2e
+ depends_on:
+ broker:
+ condition: service_healthy
+ command: bash -c "bin/pulsar-perf consume -m 100000 my-topic"
+
+ produce-perf:
+ image: apachepulsar/pulsar:3.1.1
+ networks:
+ - e2e
+ depends_on:
+ broker:
+ condition: service_healthy
+ consume-perf:
+ condition: service_started
+ command: bash -c "bin/pulsar-perf produce -m 100000 my-topic"
+
+ otel-collector:
+ image: otel/opentelemetry-collector:0.87.0
+ networks:
+ - e2e
+ command: [ "--config=/etc/otel-collector-config.yaml" ]
+ volumes:
+ - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
+ expose:
+ - 55678
+ depends_on:
+ oap:
+ condition: service_healthy
+networks:
+ e2e:
diff --git a/test/e2e-v2/cases/pulsar/e2e.yaml
b/test/e2e-v2/cases/pulsar/e2e.yaml
new file mode 100644
index 0000000000..48cb6da910
--- /dev/null
+++ b/test/e2e-v2/cases/pulsar/e2e.yaml
@@ -0,0 +1,37 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This file is used to show how to write configuration files and can be used
to test.
+
+setup:
+ env: compose
+ file: docker-compose.yml
+ timeout: 20m
+ init-system-environment: ../../script/env
+ steps:
+ - name: set PATH
+ command: export PATH=/tmp/skywalking-infra-e2e/bin:$PATH
+ - name: install yq
+ command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh yq
+ - name: install swctl
+ command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh swctl
+
+verify:
+ retry:
+ count: 60
+ interval: 3s
+ cases:
+ - includes:
+ - ./pulsar-cases.yaml
diff --git a/test/e2e-v2/cases/pulsar/expected/bookie_instance.yml
b/test/e2e-v2/cases/pulsar/expected/bookie_instance.yml
new file mode 100644
index 0000000000..6a019c1718
--- /dev/null
+++ b/test/e2e-v2/cases/pulsar/expected/bookie_instance.yml
@@ -0,0 +1,22 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{- contains . }}
+- id: {{ notEmpty .id }}
+ name: bookie:8000
+ instanceuuid: {{ notEmpty .instanceuuid }}
+ attributes: []
+ language: UNKNOWN
+ {{- end }}
diff --git a/test/e2e-v2/cases/pulsar/expected/broker_instance.yml
b/test/e2e-v2/cases/pulsar/expected/broker_instance.yml
new file mode 100644
index 0000000000..3aadfa735d
--- /dev/null
+++ b/test/e2e-v2/cases/pulsar/expected/broker_instance.yml
@@ -0,0 +1,22 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{- contains . }}
+- id: {{ notEmpty .id }}
+ name: broker:8080
+ instanceuuid: {{ notEmpty .instanceuuid }}
+ attributes: []
+ language: UNKNOWN
+ {{- end }}
diff --git
a/test/e2e-v2/cases/pulsar/expected/metrics-has-value-instance-bookie-label.yml
b/test/e2e-v2/cases/pulsar/expected/metrics-has-value-instance-bookie-label.yml
new file mode 100644
index 0000000000..07d707edd9
--- /dev/null
+++
b/test/e2e-v2/cases/pulsar/expected/metrics-has-value-instance-bookie-label.yml
@@ -0,0 +1,33 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+type: TIME_SERIES_VALUES
+results:
+ {{- contains .results }}
+ - metric:
+ labels:
+ - key: _
+ value: bookie:8000
+ values:
+ {{- contains .values }}
+ - id: {{ notEmpty .id }}
+ value: {{ .value }}
+ traceid: null
+ - id: {{ notEmpty .id }}
+ value: null
+ traceid: null
+ {{- end}}
+ {{- end}}
+error: null
\ No newline at end of file
diff --git
a/test/e2e-v2/cases/pulsar/expected/metrics-has-value-instance-broker-label.yml
b/test/e2e-v2/cases/pulsar/expected/metrics-has-value-instance-broker-label.yml
new file mode 100644
index 0000000000..f93f3cbdd2
--- /dev/null
+++
b/test/e2e-v2/cases/pulsar/expected/metrics-has-value-instance-broker-label.yml
@@ -0,0 +1,33 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+type: TIME_SERIES_VALUES
+results:
+ {{- contains .results }}
+ - metric:
+ labels:
+ - key: _
+ value: broker:8080
+ values:
+ {{- contains .values }}
+ - id: {{ notEmpty .id }}
+ value: {{ .value }}
+ traceid: null
+ - id: {{ notEmpty .id }}
+ value: null
+ traceid: null
+ {{- end}}
+ {{- end}}
+error: null
\ No newline at end of file
diff --git a/test/e2e-v2/cases/pulsar/expected/metrics-has-value.yml
b/test/e2e-v2/cases/pulsar/expected/metrics-has-value.yml
new file mode 100644
index 0000000000..dc71f56632
--- /dev/null
+++ b/test/e2e-v2/cases/pulsar/expected/metrics-has-value.yml
@@ -0,0 +1,31 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+type: TIME_SERIES_VALUES
+results:
+ {{- contains .results }}
+ - metric:
+ labels: []
+ values:
+ {{- contains .values }}
+ - id: {{ notEmpty .id }}
+ value: {{ .value }}
+ traceid: null
+ - id: {{ notEmpty .id }}
+ value: null
+ traceid: null
+ {{- end}}
+ {{- end}}
+error: null
diff --git a/test/e2e-v2/cases/pulsar/expected/service.yml
b/test/e2e-v2/cases/pulsar/expected/service.yml
new file mode 100644
index 0000000000..b51451cb43
--- /dev/null
+++ b/test/e2e-v2/cases/pulsar/expected/service.yml
@@ -0,0 +1,31 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{- contains . }}
+- id: {{ b64enc "pulsar::pulsar-cluster" }}.1
+ name: pulsar::pulsar-cluster
+ group: pulsar
+ shortname: pulsar-cluster
+ layers:
+ - PULSAR
+ normal: true
+- id: {{ b64enc "bookkeeper::pulsar-cluster" }}.1
+ name: bookkeeper::pulsar-cluster
+ group: bookkeeper
+ shortname: pulsar-cluster
+ layers:
+ - BOOKKEEPER
+ normal: true
+ {{- end }}
\ No newline at end of file
diff --git a/test/e2e-v2/cases/pulsar/otel-collector-config.yaml
b/test/e2e-v2/cases/pulsar/otel-collector-config.yaml
new file mode 100644
index 0000000000..af18c27b28
--- /dev/null
+++ b/test/e2e-v2/cases/pulsar/otel-collector-config.yaml
@@ -0,0 +1,58 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+receivers:
+ # Data sources: metrics
+ prometheus:
+ config:
+ scrape_configs:
+ - job_name: pulsar-monitoring
+ scrape_interval: 15s
+ static_configs:
+ - targets: [broker:8080]
+ relabel_configs:
+ - source_labels: [ __address__ ]
+ regex: (.+)
+ target_label: node
+ replacement: $$1
+ - job_name: bookkeeper-monitoring
+ scrape_interval: 10s
+ static_configs:
+ - targets: [bookie:8000]
+ labels:
+ cluster: pulsar-cluster
+ relabel_configs:
+ - source_labels: [ __address__ ]
+ regex: (.+)
+ target_label: node
+ replacement: $$1
+
+processors:
+ batch:
+
+exporters:
+ otlp:
+ # The OAP Server address
+ endpoint: "oap:11800"
+ tls:
+ insecure: true
+ debug:
+ verbosity: detailed
+service:
+ pipelines:
+ metrics:
+ receivers: [prometheus]
+ processors: [batch]
+ exporters: [debug,otlp]
\ No newline at end of file
diff --git a/test/e2e-v2/cases/pulsar/pulsar-cases.yaml
b/test/e2e-v2/cases/pulsar/pulsar-cases.yaml
new file mode 100644
index 0000000000..091932e751
--- /dev/null
+++ b/test/e2e-v2/cases/pulsar/pulsar-cases.yaml
@@ -0,0 +1,129 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+cases:
+ # service cases for the services and instances.
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql service ls
+ expected: expected/service.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql instance ls
--service-name=pulsar::pulsar-cluster
+ expected: expected/broker_instance.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql instance ls
--service-name=bookkeeper::pulsar-cluster
+ expected: expected/bookie_instance.yml
+ # service cases for pulsar-cluster
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_pulsar_total_topics --service-name=pulsar::pulsar-cluster
+ expected: expected/metrics-has-value-instance-broker-label.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_pulsar_total_subscriptions
--service-name=pulsar::pulsar-cluster
+ expected: expected/metrics-has-value-instance-broker-label.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_pulsar_total_producers --service-name=pulsar::pulsar-cluster
+ expected: expected/metrics-has-value-instance-broker-label.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_pulsar_total_consumers --service-name=pulsar::pulsar-cluster
+ expected: expected/metrics-has-value-instance-broker-label.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_pulsar_message_rate_in --service-name=pulsar::pulsar-cluster
+ expected: expected/metrics-has-value-instance-broker-label.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_pulsar_message_rate_out --service-name=pulsar::pulsar-cluster
+ expected: expected/metrics-has-value-instance-broker-label.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_pulsar_throughput_in --service-name=pulsar::pulsar-cluster
+ expected: expected/metrics-has-value-instance-broker-label.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_pulsar_throughput_out --service-name=pulsar::pulsar-cluster
+ expected: expected/metrics-has-value-instance-broker-label.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_pulsar_storage_size --service-name=pulsar::pulsar-cluster
+ expected: expected/metrics-has-value-instance-broker-label.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_pulsar_storage_logical_size
--service-name=pulsar::pulsar-cluster
+ expected: expected/metrics-has-value-instance-broker-label.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_pulsar_storage_write_rate
--service-name=pulsar::pulsar-cluster
+ expected: expected/metrics-has-value-instance-broker-label.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_pulsar_storage_read_rate
--service-name=pulsar::pulsar-cluster
+ expected: expected/metrics-has-value-instance-broker-label.yml
+
+ # service cases for pulsar-broker
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_pulsar_broker_active_connections
--service-name=pulsar::pulsar-cluster
+ expected: expected/metrics-has-value.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_pulsar_broker_total_connections
--service-name=pulsar::pulsar-cluster
+ expected: expected/metrics-has-value.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_pulsar_broker_connection_create_success_count
--service-name=pulsar::pulsar-cluster
+ expected: expected/metrics-has-value.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_pulsar_broker_connection_create_fail_count
--service-name=pulsar::pulsar-cluster
+ expected: expected/metrics-has-value.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_pulsar_broker_connection_closed_total_count
--service-name=pulsar::pulsar-cluster
+ expected: expected/metrics-has-value.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_pulsar_broker_jvm_memory_used
--service-name=pulsar::pulsar-cluster
+ expected: expected/metrics-has-value.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_pulsar_broker_jvm_memory_committed
--service-name=pulsar::pulsar-cluster
+ expected: expected/metrics-has-value.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_pulsar_broker_jvm_memory_init
--service-name=pulsar::pulsar-cluster
+ expected: expected/metrics-has-value.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_pulsar_broker_jvm_threads_current
--service-name=pulsar::pulsar-cluster
+ expected: expected/metrics-has-value.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_pulsar_broker_jvm_threads_daemon
--service-name=pulsar::pulsar-cluster
+ expected: expected/metrics-has-value.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_pulsar_broker_jvm_threads_peak
--service-name=pulsar::pulsar-cluster
+ expected: expected/metrics-has-value.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_pulsar_broker_jvm_threads_deadlocked
--service-name=pulsar::pulsar-cluster
+ expected: expected/metrics-has-value.yml
+
+ # service cases for the bookkeeper cluster
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_bookkeeper_bookie_ledgers_count
--service-name=bookkeeper::pulsar-cluster
+ expected: expected/metrics-has-value-instance-bookie-label.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_bookkeeper_bookie_ledger_writable_dirs
--service-name=bookkeeper::pulsar-cluster
+ expected: expected/metrics-has-value-instance-bookie-label.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_bookkeeper_bookie_ledger_dir_data_bookkeeper_ledgers_usage
--service-name=bookkeeper::pulsar-cluster
+ expected: expected/metrics-has-value-instance-bookie-label.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_bookkeeper_bookie_entries_count
--service-name=bookkeeper::pulsar-cluster
+ expected: expected/metrics-has-value-instance-bookie-label.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_bookkeeper_bookie_write_cache_size
--service-name=bookkeeper::pulsar-cluster
+ expected: expected/metrics-has-value-instance-bookie-label.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_bookkeeper_bookie_write_cache_count
--service-name=bookkeeper::pulsar-cluster
+ expected: expected/metrics-has-value-instance-bookie-label.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_bookkeeper_bookie_read_cache_size
--service-name=bookkeeper::pulsar-cluster
+ expected: expected/metrics-has-value-instance-bookie-label.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_bookkeeper_bookie_read_cache_count
--service-name=bookkeeper::pulsar-cluster
+ expected: expected/metrics-has-value-instance-bookie-label.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_bookkeeper_bookie_write_rate
--service-name=bookkeeper::pulsar-cluster
+ expected: expected/metrics-has-value-instance-bookie-label.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_bookkeeper_bookie_read_rate
--service-name=bookkeeper::pulsar-cluster
+ expected: expected/metrics-has-value-instance-bookie-label.yml
+
+
+ # service cases for the bookkeeper node
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_bookkeeper_node_thread_executor_completed
--service-name=bookkeeper::pulsar-cluster
+ expected: expected/metrics-has-value.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_bookkeeper_node_thread_executor_tasks_completed
--service-name=bookkeeper::pulsar-cluster
+ expected: expected/metrics-has-value.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_bookkeeper_node_thread_executor_tasks_rejected
--service-name=bookkeeper::pulsar-cluster
+ expected: expected/metrics-has-value.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_bookkeeper_node_thread_executor_tasks_failed
--service-name=bookkeeper::pulsar-cluster
+ expected: expected/metrics-has-value.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_bookkeeper_node_high_priority_threads
--service-name=bookkeeper::pulsar-cluster
+ expected: expected/metrics-has-value.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_bookkeeper_node_read_thread_pool_threads
--service-name=bookkeeper::pulsar-cluster
+ expected: expected/metrics-has-value.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_bookkeeper_node_high_priority_thread_max_queue_size
--service-name=bookkeeper::pulsar-cluster
+ expected: expected/metrics-has-value.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_bookkeeper_node_read_thread_pool_max_queue_size
--service-name=bookkeeper::pulsar-cluster
+ expected: expected/metrics-has-value.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_bookkeeper_node_jvm_memory_used
--service-name=bookkeeper::pulsar-cluster
+ expected: expected/metrics-has-value.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_bookkeeper_node_jvm_memory_committed
--service-name=bookkeeper::pulsar-cluster
+ expected: expected/metrics-has-value.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_bookkeeper_node_jvm_memory_init
--service-name=bookkeeper::pulsar-cluster
+ expected: expected/metrics-has-value.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_bookkeeper_node_jvm_threads_current
--service-name=bookkeeper::pulsar-cluster
+ expected: expected/metrics-has-value.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_bookkeeper_node_jvm_threads_daemon
--service-name=bookkeeper::pulsar-cluster
+ expected: expected/metrics-has-value.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_bookkeeper_node_jvm_threads_peak
--service-name=bookkeeper::pulsar-cluster
+ expected: expected/metrics-has-value.yml
+ - query: swctl --display yaml
--base-url=http://${oap_host}:${oap_12800}/graphql metrics exec
--expression=meter_bookkeeper_node_jvm_threads_deadlocked
--service-name=bookkeeper::pulsar-cluster
+ expected: expected/metrics-has-value.yml
\ No newline at end of file
