dragosvictor opened a new pull request, #21854:
URL: https://github.com/apache/pulsar/pull/21854
<!--
### Contribution Checklist
- PR title format should be *[type][component] summary*. For details, see
*[Guideline - Pulsar PR Naming
Convention](https://pulsar.apache.org/contribute/develop-semantic-title/)*.
- Fill out the template below to describe the changes contributed by the
pull request. That will give reviewers the context they need to do the review.
- Each pull request should address only one issue, not mix up code from
multiple issues.
- Each commit in the pull request has a meaningful commit message
- Once all items of the checklist are addressed, remove the above text and
this checklist, leaving only the filled out template below.
-->
<!-- If the PR belongs to a PIP, please add the PIP link here -->
PIP: [PIP-307](https://github.com/apache/pulsar/blob/master/pip/pip-307.md)
<!-- Details of when a PIP is required and how the PIP process work, please
see: https://github.com/apache/pulsar/blob/master/pip/README.md -->
### Motivation
This PR adds metrics to monitor the broker behavior during the graceful
closure of producers and consumers.
### Modifications
These new _broker_ level metrics are being proposed:
- `brk_lb_release_latency`: type summary, exposes a histogram of
milliseconds spent in the `RELEASE` state on the destination broker
- `brk_lb_assign_latency`: type summary, exposes a histogram of
milliseconds spent in the `ASSIGN` state on the source broker
- `brk_lb_unload_latency`: type summary, exposes a histogram of total time
spent (in milliseconds) unloading a topic (from state `ASSIGN` to state `FREE`
or `OWNED`) on both the source and destination brokers
- `brk_lb_ignored_ack_total`: type gauge, exposes the total number of
message ACKs ignored from consumers during topic unloading on the source broker
- `brk_lb_ignored_send_total`: type gauge, exposes the total number of
ignored messages sent by producers during topic unloading on the source broker
A sample reading of these metrics from the `/metrics/` endpoint follows.
Note that gauge metrics have their prefixes changed from `brk_` to `pulsar_`
[here](https://github.com/apache/pulsar/blob/5918efdece4386c0e4928d829832b761144fd12a/pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricsGenerator.java#L272);
for consistency, let's stick to `brk_` for now, even though this may change
soon.
```
# TYPE brk_lb_assign_latency summary
brk_lb_assign_latency{cluster="pulsar-mini",quantile="0.0"} 11.426
brk_lb_assign_latency{cluster="pulsar-mini",quantile="0.5"} 11.426
brk_lb_assign_latency{cluster="pulsar-mini",quantile="0.95"} 11.426
brk_lb_assign_latency{cluster="pulsar-mini",quantile="0.99"} 11.426
brk_lb_assign_latency{cluster="pulsar-mini",quantile="0.999"} 11.426
brk_lb_assign_latency{cluster="pulsar-mini",quantile="0.9999"} 11.426
brk_lb_assign_latency{cluster="pulsar-mini",quantile="1.0"} 11.426
brk_lb_assign_latency_count{cluster="pulsar-mini"} 5.0
brk_lb_assign_latency_sum{cluster="pulsar-mini"} 71.0
# TYPE brk_lb_unload_latency summary
brk_lb_unload_latency{cluster="pulsar-mini",quantile="0.0"} 13.784
brk_lb_unload_latency{cluster="pulsar-mini",quantile="0.5"} 48.821
brk_lb_unload_latency{cluster="pulsar-mini",quantile="0.95"} 48.821
brk_lb_unload_latency{cluster="pulsar-mini",quantile="0.99"} 48.821
brk_lb_unload_latency{cluster="pulsar-mini",quantile="0.999"} 48.821
brk_lb_unload_latency{cluster="pulsar-mini",quantile="0.9999"} 48.821
brk_lb_unload_latency{cluster="pulsar-mini",quantile="1.0"} 48.821
brk_lb_unload_latency_count{cluster="pulsar-mini"} 8.0
brk_lb_unload_latency_sum{cluster="pulsar-mini"} 342.0
# TYPE brk_lb_release_latency summary
brk_lb_release_latency{cluster="pulsar-mini",quantile="0.0"} 1.787
brk_lb_release_latency{cluster="pulsar-mini",quantile="0.5"} 1.787
brk_lb_release_latency{cluster="pulsar-mini",quantile="0.95"} 1.787
brk_lb_release_latency{cluster="pulsar-mini",quantile="0.99"} 1.787
brk_lb_release_latency{cluster="pulsar-mini",quantile="0.999"} 1.787
brk_lb_release_latency{cluster="pulsar-mini",quantile="0.9999"} 1.787
brk_lb_release_latency{cluster="pulsar-mini",quantile="1.0"} 1.787
brk_lb_release_latency_count{cluster="pulsar-mini"} 4.0
brk_lb_release_latency_sum{cluster="pulsar-mini"} 10.0
# TYPE pulsar_lb_ignored_ack_total gauge
pulsar_lb_ignored_ack_total{cluster="pulsar-mini",
broker="pulsar-mini-broker-0.pulsar-mini-broker.pulsar.svc.cluster.local",
metric="bundleReleasing"} 55
# TYPE pulsar_lb_ignored_send_total gauge
pulsar_lb_ignored_send_total{cluster="pulsar-mini",
broker="pulsar-mini-broker-0.pulsar-mini-broker.pulsar.svc.cluster.local",
metric="bundleReleasing"} 107
```
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
This change added tests and can be verified as follows:
- Modified test
`org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest#testGetMetrics`
to validate ignored (ack/send) count metric values
- Modified test
`org.apache.pulsar.broker.stats.PrometheusMetricsTest#testBundlesMetrics` to
validate the existence of new latency metrics
### Does this pull request potentially affect one of the following parts:
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
*If the box was checked, please highlight the changes*
- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [x] The metrics
- [ ] Anything that affects deployment
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update
later -->
- [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
### Matching PR in forked repository
PR in forked repository: https://github.com/dragosvictor/pulsar/pull/4
<!--
After opening this PR, the build in apache/pulsar will fail and instructions
will
be provided for opening a PR in the PR author's forked repository.
apache/pulsar pull requests should be first tested in your own fork since
the
apache/pulsar CI based on GitHub Actions has constrained resources and quota.
GitHub Actions provides separate quota for pull requests that are executed
in
a forked repository.
The tests will be run in the forked repository until all PR review comments
have
been handled, the tests pass and the PR is approved by a reviewer.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
