janhoy opened a new pull request, #5236: URL: https://github.com/apache/solr/pull/5236
https://issues.apache.org/jira/browse/SOLR-18535 Replaces the placeholder alert rule from SOLR-18533 with a real rule set, generated from the jsonnet mixin into `solr/monitoring/prometheus-solr-alerts.yml`. One group, `SolrAlerts`, with a threshold setting per rule in `mixin/config.libsonnet`. | Alert | Severity | Fires when | Setting | |---|---|---|---| | `SolrHighHeapUsage` | critical | Heap use above 90% for 2m | `heapUsageThreshold` | | `SolrJvmGcThrashing` | critical | Over 10 seconds of GC per minute for 3m | `gcSecondsPerMinuteThreshold` | | `SolrLowDiskSpace` | critical | Under 15% free on the Solr home partition for 5m | `diskFreeThreshold` | | `SolrNodeDown` | warning | Scrapes failing for 2m | — | | `SolrHighSearchLatency` | warning | `/select` p99 above 1000 ms for 5m | `searchLatencyThresholdMs` | | `SolrHighErrorRate` | warning | Over 1% of requests failing for 5m | `errorRateThreshold` | | `SolrOverseerQueueBuildup` | warning | Over 50 queued collection operations for 5m (SolrCloud only) | `overseerQueueThreshold` | | `SolrHighMmapRatio` | warning | Under 10% of the index fits in memory outside the heap for 5m | `mmapRatioThreshold` | Ported from the draft in #4210, with two fixes worth calling out: * **Heap usage** compared the largest pool's usage against the largest pool's limit. The OTel JVM metrics do not work that way: under G1 only `G1 Old Gen` reports `jvm_memory_limit_bytes` (the whole `-Xmx`), while every pool reports its own usage. On a live node the draft expression read 41% where real usage was 58%. It now sums per-pool usage over per-pool limits, the same shape the dashboard uses. * **GC thrashing** compared a `rate()`, in seconds per second, against a threshold documented in seconds per minute — 10 s/s is unreachable, so the rule could never fire. The expression now scales by 60, so the setting means what its name says. Smaller changes: every expression now carries `solrSelector` (four rules in the draft did not, so they would also match JVM metrics from other Java services), `rate()` windows come from the existing `rateInterval` setting instead of a hardcoded `1m`, and the latency annotation no longer runs `humanizeDuration` over a millisecond value. Docs: the reference guide's Alert Rules section lists the rules and the setting that tunes each, and its work-in-progress note now covers only the dashboard. `solr/monitoring/README.md` gets the same table against the underlying metric names. ## Verification Against a local Solr 11 node with a `techproducts` collection, scraped by Prometheus 3.15: * Every metric and label the rules use exists in the native `/admin/metrics?wt=prometheus` output. * All 8 expressions evaluate with `health=ok` and return series: heap 0.58, GC 0.007 s/min, disk 0.090, `/select` p99 2.75 ms, error rate 0.111, Overseer 0, mmap 100%. * Three rules were observed doing their job: `SolrLowDiskSpace` fired (test machine at 91% full), `SolrHighErrorRate` went pending at 11% after 5 bad queries among 45, and `up{job="solr"} == 0` matched after stopping the node. Annotations rendered as expected. * `promtool check rules` passes; `jsonnetfmt --test` clean; config overrides via `(import 'mixin.libsonnet') + { _config+:: {...} }` reach the new thresholds. The dashboard is untouched and still the placeholder — that is SOLR-18534. 🤖 Written by Claude, checked by me -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
