Copilot commented on code in PR #5236:
URL: https://github.com/apache/solr/pull/5236#discussion_r4380704847
##########
solr/monitoring/mixin/alerts/alerts.libsonnet:
##########
@@ -41,6 +123,120 @@ local alertsFor(cfg) = {
|||,
},
},
+
+ // internal="false" drops the shard requests a distributed search fans
+ // out, which would otherwise be counted alongside the request that
+ // issued them.
+ {
+ alert: 'SolrHighSearchLatency',
+ expr: |||
+ histogram_quantile(0.99,
+ sum by (le, instance) (
+ rate(
+
solr_core_requests_times_milliseconds_bucket{%(solrSelector)s,handler=~"/select.*",internal="false"}[%(rateInterval)s]
+ )
+ )
+ )
+ > %(threshold)s
+ ||| % {
+ threshold: cfg.searchLatencyThresholdMs,
+ solrSelector: cfg.solrSelector,
+ rateInterval: cfg.rateInterval,
+ },
+ 'for': '5m',
+ labels: { severity: 'warning' },
+ annotations: {
+ summary: 'Solr node {{ $labels.instance }} has high search
latency',
+ description: |||
+ Node {{ $labels.instance }} answers 99%% of searches in {{
$value | humanize }}ms
+ (threshold: %(threshold)sms). Usual causes are expensive queries
or faceting, caches
+ too small for the working set, and GC pauses.
+ ||| % { threshold: cfg.searchLatencyThresholdMs },
+ },
+ },
+
+ // Counts client (4xx) and server (5xx) failures alike; add
+ // source="server" to the errors selector to ignore malformed requests.
+ {
+ alert: 'SolrHighErrorRate',
+ expr: |||
+ sum by (instance)
(rate(solr_core_requests_errors_total{%(solrSelector)s}[%(rateInterval)s]))
+ /
+ sum by (instance)
(rate(solr_core_requests_total{%(solrSelector)s}[%(rateInterval)s]))
Review Comment:
These selectors count both core counters and optional node rollups emitted
under the same names by `AttributedInstrumentFactory`. When
`aggregateNodeLevelMetricsEnabled` is enabled for only some handlers or cores,
the uneven duplication can suppress the error-rate alert or distort p99 in the
latency rule at line 136. Add `core!=""` to both counters and the latency
histogram selector, then regenerate `prometheus-solr-alerts.yml`.
##########
solr/monitoring/mixin/alerts/alerts.libsonnet:
##########
@@ -26,8 +26,90 @@
local alertsFor(cfg) = {
groups: [
{
- name: 'SolrExampleAlerts',
+ name: 'SolrAlerts',
rules: [
+
+ // The inner max collapses pools reported by more than one OTel scope,
+ // which differ by Solr version. Pools are summed on both sides: under
+ // G1 only one pool reports a limit, and that limit is the whole -Xmx.
+ {
+ alert: 'SolrHighHeapUsage',
+ expr: |||
+ sum by (instance) (
+ max by (instance, jvm_memory_pool_name) (
+ jvm_memory_used_bytes{%(solrSelector)s,jvm_memory_type="heap"}
+ )
+ )
+ /
+ sum by (instance) (
+ max by (instance, jvm_memory_pool_name) (
+ jvm_memory_limit_bytes{%(solrSelector)s,jvm_memory_type="heap"}
Review Comment:
Generational ZGC reports the full `-Xmx` as the limit of both young and old
pools, so summing them halves the calculated heap usage and prevents the
default 90% alert from firing. The same calculation at lines 213–215 subtracts
the heap twice in `SolrHighMmapRatio`, potentially producing false alerts. Use
a collector-aware whole-heap maximum in both rules and regenerate
`prometheus-solr-alerts.yml`.
##########
solr/monitoring/mixin/alerts/alerts.libsonnet:
##########
@@ -41,6 +123,120 @@ local alertsFor(cfg) = {
|||,
},
},
+
+ // internal="false" drops the shard requests a distributed search fans
+ // out, which would otherwise be counted alongside the request that
+ // issued them.
+ {
+ alert: 'SolrHighSearchLatency',
+ expr: |||
+ histogram_quantile(0.99,
+ sum by (le, instance) (
+ rate(
+
solr_core_requests_times_milliseconds_bucket{%(solrSelector)s,handler=~"/select.*",internal="false"}[%(rateInterval)s]
+ )
+ )
+ )
+ > %(threshold)s
+ ||| % {
+ threshold: cfg.searchLatencyThresholdMs,
+ solrSelector: cfg.solrSelector,
+ rateInterval: cfg.rateInterval,
+ },
+ 'for': '5m',
+ labels: { severity: 'warning' },
+ annotations: {
+ summary: 'Solr node {{ $labels.instance }} has high search
latency',
+ description: |||
+ Node {{ $labels.instance }} answers 99%% of searches in {{
$value | humanize }}ms
+ (threshold: %(threshold)sms). Usual causes are expensive queries
or faceting, caches
+ too small for the working set, and GC pauses.
+ ||| % { threshold: cfg.searchLatencyThresholdMs },
+ },
+ },
+
+ // Counts client (4xx) and server (5xx) failures alike; add
+ // source="server" to the errors selector to ignore malformed requests.
+ {
+ alert: 'SolrHighErrorRate',
+ expr: |||
+ sum by (instance)
(rate(solr_core_requests_errors_total{%(solrSelector)s}[%(rateInterval)s]))
+ /
+ sum by (instance)
(rate(solr_core_requests_total{%(solrSelector)s}[%(rateInterval)s]))
+ > %(threshold)g
+ ||| % {
+ threshold: cfg.errorRateThreshold,
+ solrSelector: cfg.solrSelector,
+ rateInterval: cfg.rateInterval,
+ },
+ 'for': '5m',
+ labels: { severity: 'warning' },
+ annotations: {
+ summary: 'Solr node {{ $labels.instance }} has a high request
error rate',
+ description: |||
+ {{ $value | humanizePercentage }} of requests to {{
$labels.instance }} are failing
+ (threshold: %(threshold)s%%). The Solr log carries the cause:
invalid queries,
+ missing fields, cores that failed to load, or connectivity
problems.
+ ||| % { threshold: std.floor(cfg.errorRateThreshold * 100) },
+ },
+ },
+
+ // SolrCloud only; silent on a standalone node.
+ {
+ alert: 'SolrOverseerQueueBuildup',
+ expr: |||
+ max by (instance)
(solr_overseer_collection_work_queue_size{%(solrSelector)s})
+ > %(threshold)s
+ ||| % { threshold: cfg.overseerQueueThreshold, solrSelector:
cfg.solrSelector },
+ 'for': '5m',
+ labels: { severity: 'warning' },
+ annotations: {
+ summary: 'Solr Overseer collection work queue is building up on {{
$labels.instance }}',
+ description: |||
+ {{ $value }} collection operations are queued on {{
$labels.instance }}
+ (threshold: %(threshold)s). The Overseer is falling behind: look
for long-running
+ collection operations, a disrupted election, or slow ZooKeeper.
+ ||| % { threshold: cfg.overseerQueueThreshold },
+ },
+ },
+
+ // How much of the index fits in the memory left for the page cache.
+ // The trailing guard keeps the rule silent when
jvm_system_memory_bytes
+ // is missing or zero, which would make available memory look negative.
+ {
+ alert: 'SolrHighMmapRatio',
+ expr: |||
+ (
+ clamp_max(
+ (
+ max by (instance)
(jvm_system_memory_bytes{%(solrSelector)s,state="total"})
+ - sum by (instance) (
+ max by (instance, jvm_memory_pool_name) (
+
jvm_memory_limit_bytes{%(solrSelector)s,jvm_memory_type="heap"}
+ )
+ )
+ )
+ / 1e6
Review Comment:
The index-size gauge is emitted by SolrCore using
`MetricUtils.bytesToMegabytes()`, which divides bytes by `1024 * 1024`, not
`1e6`. This conversion therefore inflates the estimated fit percentage by about
4.86%: an actual 9.8% becomes 10.28% and incorrectly suppresses the default
alert. Use the same binary conversion here and regenerate
`prometheus-solr-alerts.yml` (line 150).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
