gortiz opened a new pull request, #19784: URL: https://github.com/apache/pinot/pull/19784
## Summary Part of #19782. This PR makes the cost of routing rebuilds visible per table. `BaseBrokerRoutingManager.processAssignmentChangeForTable` measures the time to read the ideal state and the external view (IS/EV) and to run `RoutingEntry.onAssignmentChange`. On success, it records the timer and increments `ROUTING_TABLE_UPDATES`. On an exception, it increments `ROUTING_TABLE_UPDATE_FAILURES` instead. In both cases, it sets the segment gauge. ## Metrics | Metric | Enum | Type | Scope | Value | |---|---|---|---|---| | `routingTableUpdateTime` | `BrokerTimer.ROUTING_TABLE_UPDATE_TIME` | timer | per table and global | Time to read the IS/EV and rebuild the routing components. Successful rebuilds only. | | `routingTableUpdates` | `BrokerMeter.ROUTING_TABLE_UPDATES` (new) | meter | per table | Number of successful rebuilds after an IS/EV change. | | `routingTableUpdateFailures` | `BrokerMeter.ROUTING_TABLE_UPDATE_FAILURES` (new) | meter | per table | Number of failed rebuilds. The routing entry keeps the old IS/EV versions, so the next change tries again. | | `routingTableSegments` | `BrokerGauge.ROUTING_TABLE_SEGMENTS` (new) | gauge | per table | Number of online segments of the ideal state that the segment pre-selector keeps. | `ROUTING_TABLE_UPDATE_TIME` already existed and had a JMX exporter rule, but no code emitted it. Details to check: - A failed rebuild is not timed and not counted in `routingTableUpdates`. Thus a table that fails again and again does not look like a table with many changes. - Only rebuilds that an IS/EV change starts emit these metrics. A full routing build (for example, after a table config change) does not. - The gauge gets a value on the routing build and on each assignment change. The broker removes it together with the replica health gauges in `removeRouting`. The broker reports it only after it stores the routing entry, so a failed build cannot leave a gauge behind. - `RoutingEntry` keeps the segment count in a new field. Only the thread that holds the build lock of the table writes and reads it. ## Concurrency The query path does not change. The metric calls run in the rebuild thread, after the rebuild. ## Compatibility - The name `routingTableUpdateTime` does not change. The existing rule in `pinot.yml` (`pinot_broker_routingTableUpdateTime_*`) matches the global timer, which now gets values. - The catch-all table metric rules in `broker.yml` and `pinot.yml` export the new names. Thus this PR adds no exporter rules, like other recent broker metric PRs. - There are no config, API, or wire format changes. ## Tests - `BrokerRoutingManagerTest.testAssignmentChangeReportsRebuildMetrics`: an assignment change records the per-table and global timer, increments `ROUTING_TABLE_UPDATES` by 1, and sets the gauge to the number of online segments (two ONLINE and one OFFLINE segment in the test). - `BrokerRoutingManagerTest.testFailedAssignmentChangeReportsFailureMetric`: when the instance selector throws, the broker increments `ROUTING_TABLE_UPDATE_FAILURES` and does not increment `ROUTING_TABLE_UPDATES` or record the timer. - The existing `BrokerRoutingManagerTest` tests now also make sure that the build sets the gauge and that `removeRouting` removes it. - `BrokerPrometheusMetricsTest` (Yammer and Dropwizard) makes sure that the JMX exporter config exports the new metrics with the table labels. Local results: `org.apache.pinot.broker.routing` (18 classes) runs 254 tests with 0 failures. `YammerBrokerPrometheusMetricsTest` runs 228 tests with 0 failures. `DropwizardBrokerPrometheusMetricsTest` is disabled on master. Spotless, license and checkstyle checks pass for `pinot-common` and `pinot-broker`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
