This is an automated email from the ASF dual-hosted git repository.
janhoy pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/solr.git
The following commit(s) were added to refs/heads/main by this push:
new 13c693d805f SOLR-18533: Developer tooling for monitoring (#5109)
13c693d805f is described below
commit 13c693d805f6541f7cec684cd27d28eef7b3b614
Author: Jan Høydahl <[email protected]>
AuthorDate: Sat Oct 24 16:15:17 2026 +0200
SOLR-18533: Developer tooling for monitoring (#5109)
---
.gitattributes | 7 +
.../SOLR-18533-monitoring-dev-tooling.yml | 14 +
solr/monitoring/README.md | 203 ++++++++++++++
solr/monitoring/dev/alertmanager/alertmanager.yml | 86 ++++++
solr/monitoring/dev/docker-compose.yml | 155 +++++++++++
.../grafana/provisioning/dashboards/dashboards.yml | 28 ++
.../provisioning/datasources/prometheus.yml | 29 ++
.../monitoring/dev/prometheus/prometheus-local.yml | 48 ++++
solr/monitoring/dev/prometheus/prometheus.yml | 43 +++
solr/monitoring/dev/stack.sh | 130 +++++++++
solr/monitoring/dev/trafficgen/Dockerfile | 29 ++
solr/monitoring/dev/trafficgen/requirements.txt | 3 +
solr/monitoring/dev/trafficgen/trafficgen.py | 298 +++++++++++++++++++++
solr/monitoring/dev/trafficgen/trafficgen.yml | 45 ++++
solr/monitoring/grafana-solr-dashboard.json | 149 +++++++++++
solr/monitoring/mixin/.gitignore | 2 +
solr/monitoring/mixin/Dockerfile | 55 ++++
solr/monitoring/mixin/Makefile | 86 ++++++
solr/monitoring/mixin/alerts/alerts.libsonnet | 51 ++++
solr/monitoring/mixin/config.libsonnet | 35 +++
.../mixin/dashboards/dashboards.libsonnet | 102 +++++++
solr/monitoring/mixin/jsonnetfile.json | 15 ++
solr/monitoring/mixin/jsonnetfile.lock.json | 46 ++++
solr/monitoring/mixin/license-header.yml | 17 ++
solr/monitoring/mixin/make.sh | 67 +++++
solr/monitoring/mixin/mixin.libsonnet | 36 +++
solr/monitoring/prometheus-solr-alerts.yml | 30 +++
.../modules/deployment-guide/deployment-nav.adoc | 1 +
.../grafana-solr-dashboard.png | Bin 808611 -> 42073 bytes
.../prometheus-solr-ping.png | Bin 425471 -> 0 bytes
.../solr-exporter-diagram.png | Bin 29070 -> 0 bytes
.../solr-monitoring-diagram.png | Bin 0 -> 73028 bytes
.../solr-monitoring-diagram.svg | Bin 0 -> 9482 bytes
.../deployment-guide/pages/metrics-reporting.adoc | 2 +
.../monitoring-with-prometheus-and-grafana.adoc | 171 ++++++++++++
35 files changed, 1983 insertions(+)
diff --git a/.gitattributes b/.gitattributes
index 569be426630..8b238566943 100644
--- a/.gitattributes
+++ b/.gitattributes
@@ -9,3 +9,10 @@ versions.lock text eol=lf
/.* export-ignore
/dev-docs export-ignore
AGENTS.md export-ignore
+
+# Monitoring mixin generated outputs — always LF to ensure make regeneration
is idempotent.
+solr/monitoring/grafana-solr-dashboard.json text eol=lf
+solr/monitoring/prometheus-solr-alerts.yml text eol=lf
+
+# Contributor-only tooling, excluded from the source release like dev-docs.
+solr/monitoring/dev export-ignore
diff --git a/changelog/unreleased/SOLR-18533-monitoring-dev-tooling.yml
b/changelog/unreleased/SOLR-18533-monitoring-dev-tooling.yml
new file mode 100644
index 00000000000..33d0f76a65e
--- /dev/null
+++ b/changelog/unreleased/SOLR-18533-monitoring-dev-tooling.yml
@@ -0,0 +1,14 @@
+# See https://github.com/apache/solr/blob/main/dev-docs/changelog.adoc
+
+title: >
+ Add tooling for building Solr's Grafana dashboard and Prometheus alert rules
from jsonnet sources,
+ under `solr/monitoring`, together with a docker-compose stack for testing
them locally. The
+ dashboard and alert rules shipped so far are placeholders: the real
dashboard follows in
+ SOLR-18534 and the real alert rules in SOLR-18535.
+type: other
+authors:
+ - name: Jan Høydahl
+ url: https://home.apache.org/phonebook.html?uid=janhoy
+links:
+ - name: SOLR-18533
+ url: https://issues.apache.org/jira/browse/SOLR-18533
diff --git a/solr/monitoring/README.md b/solr/monitoring/README.md
new file mode 100644
index 00000000000..b7131b0a997
--- /dev/null
+++ b/solr/monitoring/README.md
@@ -0,0 +1,203 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+# Solr Monitoring
+
+This directory provides two ready-to-use monitoring artifacts for Solr that
you can drop into
+your existing Prometheus + Grafana installation:
+
+| File | Description |
+|---|---|
+| **`grafana-solr-dashboard.json`** | Grafana dashboard — import directly into
Grafana |
+| **`prometheus-solr-alerts.yml`** | Prometheus alert rules — reference from
your `prometheus.yml` |
+
+These are the **main artifacts**. Everything else in this directory supports
their creation
+or testing.
+
+| File / Directory | Description |
+|---|---|
+| `mixin/` | Jsonnet source (single source of truth used to regenerate the
artifacts above) |
+| `dev/` | **Developer convenience only** — docker-compose stack for testing
changes to the artifacts |
+
+> **Both artifacts are placeholders right now** — a two-panel dashboard and a
single alert, which
+> exist to prove the build pipeline works end to end. The real dashboard
arrives in
+> [SOLR-18534](https://issues.apache.org/jira/browse/SOLR-18534), the alert
rules in
+> [SOLR-18535](https://issues.apache.org/jira/browse/SOLR-18535), and OTLP
push support in
+> [SOLR-18536](https://issues.apache.org/jira/browse/SOLR-18536).
+
+For the full integration guide see the Solr Reference Guide:
+**[Monitoring Solr with Prometheus and
Grafana](../solr-ref-guide/modules/deployment-guide/pages/monitoring-with-prometheus-and-grafana.adoc)**
+
+---
+
+## Importing the Dashboard
+
+1. Open Grafana → **Dashboards → Import**
+2. Upload `grafana-solr-dashboard.json` or paste its contents
+3. Select your Prometheus datasource when prompted
+4. Click **Import**
+
+The placeholder dashboard has two panels, Request Rate and Heap Used, and two
dropdowns:
+`datasource` and `interval`.
+
+---
+
+## Loading Alert Rules into Prometheus
+
+Add the alert rules file to your Prometheus configuration:
+
+```yaml
+# prometheus.yml
+rule_files:
+ - /path/to/prometheus-solr-alerts.yml
+```
+
+Validate locally before deploying:
+
+```bash
+promtool check rules prometheus-solr-alerts.yml
+```
+
+One placeholder rule is included: `SolrNodeDown`, which fires when a Solr
target has failed to
+scrape for 2 minutes.
+
+---
+
+## Customizing
+
+All tunable settings live in `mixin/config.libsonnet`:
+
+```jsonnet
+{
+ _config:: {
+ solrSelector: 'job="solr"', // label selector identifying Solr targets
+ rateInterval: '5m', // window for rate() queries
+ },
+}
+```
+
+`solrSelector` must match your Prometheus scrape job, or no panel or rule will
match a series.
+Edit the file and regenerate, or patch `_config` from your own jsonnet without
touching this tree:
+
+```jsonnet
+(import 'mixin.libsonnet') + { _config+:: { solrSelector: 'job="my-solr"' } }
+```
+
+Every key in `config.libsonnet` is consumed by the dashboard or the alert
rules; the file grows as
+they do.
+
+---
+
+## Mixin Build
+
+Only contributors who need to **regenerate** the artifacts need build tooling.
+Operators can use the pre-generated files directly.
+
+There are two equivalent ways to build:
+
+### Option A — Docker (no local tool installation required)
+
+```bash
+cd mixin
+./make.sh install # fetch grafonnet dependency into vendor/
+./make.sh all # regenerate both artifacts
+./make.sh check # validate alert rules
+```
+
+`make.sh` automatically builds the `solr-mixin-make:latest` Docker image on
first run, with all
+tool versions pinned in `mixin/Dockerfile`. Pass `--rebuild` to force a fresh
image build
+(e.g. after a Dockerfile update).
+
+### Option B — Local tools
+
+Install the required tools once:
+
+| Tool | Purpose | Install |
+|---|---|---|
+| `jsonnet` (go-jsonnet) | Evaluate jsonnet source | `brew install go-jsonnet`
or `go install github.com/google/go-jsonnet/cmd/jsonnet@latest` |
+| `jb` (jsonnet-bundler) | Manage dependencies | `brew install
jsonnet-bundler` or `go install
github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@latest` |
+| `gojsontoyaml` | Convert JSON→YAML for alert rules | `go install
github.com/brancz/gojsontoyaml@latest` |
+| `promtool` *(optional)* | Validate alert rules | `brew install prometheus` |
+
+Then build:
+
+```bash
+cd mixin
+make install # fetch grafonnet dependency into vendor/
+make all # regenerate both artifacts
+```
+
+A target aborts before touching any output if a tool is missing, and writes
through a temp file,
+so a failed build cannot leave a half-written artifact behind.
+
+### Makefile targets
+
+Both `./make.sh <target>` and `make <target>` accept the same targets:
+
+| Target | Action |
+|---|---|
+| `install` | Download jsonnet dependencies (grafonnet) into `vendor/` |
+| `dashboards` | Regenerate `grafana-solr-dashboard.json` |
+| `alerts` | Regenerate `prometheus-solr-alerts.yml` |
+| `all` | Regenerate both outputs |
+| `check` | Run `promtool check rules` on alert rules |
+| `fmt` | Auto-format all `.libsonnet` source files |
+
+`vendor/` is not checked in; `jsonnetfile.lock.json` pins the exact grafonnet
revision, so
+`make install` reproduces the same dependency tree.
+
+**Commit the regenerated artifacts together with the jsonnet change that
produced them** —
+nothing in the Gradle build regenerates or verifies them.
+
+---
+
+## Developer Convenience Stack
+
+The `dev/` directory contains a docker-compose stack that starts a local Solr
+ Prometheus
++ Grafana + Alertmanager environment. **This is solely for contributors who
want to
+visually test changes to the two artifacts above.** It is not a reference
deployment
+or a recommended production setup, and is excluded from the source release.
+
+```bash
+cd dev
+./stack.sh # bundled Solr nodes + monitoring + traffic
generator
+./stack.sh --local-solr # monitoring only, scraping a Solr you already run
on :8983
+./stack.sh --down # stop and remove everything
+```
+
+Use `stack.sh` rather than `docker compose up` directly: the bundled Solr
nodes sit behind the
+`bundled-solr` compose profile, so a plain `docker compose up` would start the
monitoring services
+with no Solr to scrape.
+
+Services: Solr (`:8983`, `:8984`), Prometheus (`:9090`), Grafana (`:3000`),
Alertmanager
+(`:9093`). Grafana opens straight on the provisioned dashboard with no login —
anonymous viewing
+is enabled because this stack is local and throwaway; sign in as
`admin`/`admin` to edit. The
+Prometheus datasource is provisioned too, and Prometheus loads the alert
rules. Both artifacts are bind-mounted as single
+files, which the containers resolve once at start, so after regenerating one,
recreate them:
+
+```bash
+docker compose up -d --force-recreate grafana prometheus
+```
+
+The monitoring images are pinned for reproducibility. Override `SOLR_IMAGE`,
`PROMETHEUS_IMAGE`,
+`GRAFANA_IMAGE` or `ALERTMANAGER_IMAGE` to test against other versions:
+
+```bash
+GRAFANA_IMAGE=grafana/grafana:latest ./stack.sh
+```
+
+`trafficgen/` is a small Python container that indexes and queries
continuously, so the panels
+show movement. It exists for this stack only.
diff --git a/solr/monitoring/dev/alertmanager/alertmanager.yml
b/solr/monitoring/dev/alertmanager/alertmanager.yml
new file mode 100644
index 00000000000..dd475ba305f
--- /dev/null
+++ b/solr/monitoring/dev/alertmanager/alertmanager.yml
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# alertmanager.yml — Example Alertmanager routing configuration for Solr
alerts.
+#
+# Two-tier severity routing:
+# severity: critical → critical-receiver (intended for: PagerDuty,
OpsGenie, paging on-call)
+# severity: warning → warning-receiver (intended for: Slack, email,
non-urgent notification)
+#
+# Replace the stub "null" receivers with your real notification integrations.
+# See https://prometheus.io/docs/alerting/latest/configuration/ for receiver
examples.
+
+global:
+ resolve_timeout: 5m
+
+route:
+ # Default receiver for any alert not matched by child routes.
+ receiver: "default"
+ group_by: ["alertname", "instance"]
+ group_wait: 30s
+ group_interval: 5m
+ repeat_interval: 4h
+
+ routes:
+ # Critical alerts — page the on-call engineer immediately.
+ # Solr critical alerts: SolrHighHeapUsage, SolrJvmGcThrashing,
SolrLowDiskSpace
+ - matchers:
+ - severity = "critical"
+ receiver: "critical-receiver"
+ group_wait: 10s
+ repeat_interval: 1h
+
+ # Warning alerts — notify the team but do not page.
+ # Solr warning alerts: SolrHighSearchLatency, SolrHighErrorRate,
+ # SolrOverseerQueueBuildup, SolrHighMmapRatio
+ - matchers:
+ - severity = "warning"
+ receiver: "warning-receiver"
+ repeat_interval: 6h
+
+receivers:
+ # Default catch-all receiver (no-op stub).
+ - name: "default"
+
+ # Critical receiver — replace with PagerDuty, OpsGenie, VictorOps, or
similar.
+ # PagerDuty example:
+ # - name: "critical-receiver"
+ # pagerduty_configs:
+ # - service_key: YOUR_PAGERDUTY_INTEGRATION_KEY
+ - name: "critical-receiver"
+
+ # Warning receiver — replace with Slack, email, or similar.
+ # Slack example:
+ # - name: "warning-receiver"
+ # slack_configs:
+ # - api_url: https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK
+ # channel: '#solr-alerts'
+ # title: '{{ .GroupLabels.alertname }}'
+ # text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'
+ # Email example:
+ # - name: "warning-receiver"
+ # email_configs:
+ # - to: '[email protected]'
+ # from: '[email protected]'
+ # smarthost: 'smtp.example.com:587'
+ - name: "warning-receiver"
+
+inhibit_rules:
+ # Suppress warning alerts on an instance when a critical alert is already
firing for it.
+ - source_matchers:
+ - severity = "critical"
+ target_matchers:
+ - severity = "warning"
+ equal: ["instance"]
diff --git a/solr/monitoring/dev/docker-compose.yml
b/solr/monitoring/dev/docker-compose.yml
new file mode 100644
index 00000000000..4af98d4691b
--- /dev/null
+++ b/solr/monitoring/dev/docker-compose.yml
@@ -0,0 +1,155 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# ============================================================================
+# DEVELOPER CONVENIENCE ONLY — NOT FOR PRODUCTION USE
+# ============================================================================
+#
+# This docker-compose stack is provided solely to make it easy for contributors
+# to view and test changes to the two generated monitoring artifacts:
+#
+# ../grafana-solr-dashboard.json — Grafana dashboard
+# ../prometheus-solr-alerts.yml — Prometheus alert rules
+#
+# It is NOT intended as a reference deployment or a recommended production
setup.
+# For integrating the artifacts into your own Prometheus + Grafana
installation,
+# see the Solr Reference Guide (monitoring-with-prometheus-and-grafana.adoc).
+#
+# Quick start:
+# cd solr/monitoring/dev
+# ./stack.sh # Full stack with bundled Solr nodes
+# ./stack.sh --local-solr # Monitoring only — point at your
locally-running Solr
+# ./stack.sh --help # All options
+#
+# Ports are published on loopback only: the stack runs Solr and ZooKeeper
without
+# authentication, so it must not be reachable from other hosts on the network.
+#
+# Services (full stack):
+# solr1 http://localhost:8983 (SolrCloud node 1, embedded ZooKeeper)
+# solr2 http://localhost:8984 (SolrCloud node 2)
+# Prometheus http://localhost:9090
+# Grafana http://localhost:3000 (admin / admin)
+# Alertmanager http://localhost:9093
+# trafficgen (no port — continuously indexes/searches the 'test'
collection)
+#
+# Environment variables (set by stack.sh, or override manually):
+# PROMETHEUS_CONFIG Path to the prometheus config to mount (default:
./prometheus/prometheus.yml)
+# SOLR_URL Solr base URL for trafficgen (default:
http://solr1:8983/solr)
+# SOLR_IMAGE Solr image for the bundled nodes (default: solr:10)
+# PROMETHEUS_IMAGE Prometheus image (default: pinned, see below)
+# GRAFANA_IMAGE Grafana image (default: pinned, see below)
+# ALERTMANAGER_IMAGE Alertmanager image (default: pinned, see below)
+#
+# The monitoring images are pinned so a stack that worked yesterday still works
+# today; override any of them to test against a newer release.
+# ============================================================================
+
+services:
+ solr1:
+ image: ${SOLR_IMAGE:-solr:10}
+ profiles: [bundled-solr]
+ ports:
+ - "127.0.0.1:8983:8983"
+ - "127.0.0.1:9983:9983" # embedded ZooKeeper (Solr 10 default: ZK_PORT
= SOLR_PORT + 1000)
+ # No ZK_HOST set: Solr 10 defaults to SolrCloud with embedded ZooKeeper on
port 9983.
+ healthcheck:
+ test: ["CMD-SHELL", "curl -sf
http://localhost:8983/solr/admin/info/system || exit 1"]
+ interval: 10s
+ timeout: 5s
+ retries: 12
+ start_period: 40s
+
+ solr2:
+ image: ${SOLR_IMAGE:-solr:10}
+ profiles: [bundled-solr]
+ ports:
+ - "127.0.0.1:8984:8983"
+ environment:
+ ZK_HOST: "solr1:9983"
+ depends_on:
+ solr1:
+ condition: service_healthy
+ healthcheck:
+ test: ["CMD-SHELL", "curl -sf
http://localhost:8983/solr/admin/info/system || exit 1"]
+ interval: 10s
+ timeout: 5s
+ retries: 12
+ start_period: 30s
+
+ prometheus:
+ image: ${PROMETHEUS_IMAGE:-prom/prometheus:v3.15.0}
+ ports:
+ - "127.0.0.1:9090:9090"
+ extra_hosts:
+ - "host.docker.internal:host-gateway"
+ volumes:
+ -
${PROMETHEUS_CONFIG:-./prometheus/prometheus.yml}:/etc/prometheus/prometheus.yml:ro
+ - ../prometheus-solr-alerts.yml:/etc/prometheus/rules/solr-alerts.yml:ro
+ command:
+ - "--config.file=/etc/prometheus/prometheus.yml"
+ - "--storage.tsdb.retention.time=7d"
+ - "--web.enable-lifecycle"
+ depends_on:
+ solr1:
+ condition: service_healthy
+ required: false
+ solr2:
+ condition: service_healthy
+ required: false
+ restart: unless-stopped
+
+ grafana:
+ image: ${GRAFANA_IMAGE:-grafana/grafana:13.2.3}
+ ports:
+ - "127.0.0.1:3000:3000"
+ environment:
+ GF_SECURITY_ADMIN_USER: admin
+ GF_SECURITY_ADMIN_PASSWORD: admin
+ GF_USERS_ALLOW_SIGN_UP: "false"
+ # Local throwaway stack: let anyone view without logging in, so the
dashboard
+ # is one click away. Log in as admin/admin to edit.
+ GF_AUTH_ANONYMOUS_ENABLED: "true"
+ GF_AUTH_ANONYMOUS_ORG_ROLE: Viewer
+ # Land on the provisioned dashboard instead of Grafana's welcome page.
+ GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH:
/var/lib/grafana/dashboards/grafana-solr-dashboard.json
+ volumes:
+ - ./grafana/provisioning:/etc/grafana/provisioning:ro
+ -
../grafana-solr-dashboard.json:/var/lib/grafana/dashboards/grafana-solr-dashboard.json:ro
+ depends_on:
+ - prometheus
+ restart: unless-stopped
+
+ alertmanager:
+ image: ${ALERTMANAGER_IMAGE:-prom/alertmanager:v0.34.1}
+ ports:
+ - "127.0.0.1:9093:9093"
+ volumes:
+ - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
+ restart: unless-stopped
+
+ trafficgen:
+ build: ./trafficgen
+ extra_hosts:
+ - "host.docker.internal:host-gateway"
+ environment:
+ SOLR_URL: ${SOLR_URL:-http://solr1:8983/solr}
+ depends_on:
+ solr1:
+ condition: service_healthy
+ required: false
+ solr2:
+ condition: service_healthy
+ required: false
+ restart: unless-stopped
diff --git a/solr/monitoring/dev/grafana/provisioning/dashboards/dashboards.yml
b/solr/monitoring/dev/grafana/provisioning/dashboards/dashboards.yml
new file mode 100644
index 00000000000..8fab25f256f
--- /dev/null
+++ b/solr/monitoring/dev/grafana/provisioning/dashboards/dashboards.yml
@@ -0,0 +1,28 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Grafana dashboard provisioning — tells Grafana to load dashboards from the
+# /var/lib/grafana/dashboards directory (mounted by docker-compose.yml).
+
+apiVersion: 1
+
+providers:
+ - name: Solr
+ folder: Solr
+ type: file
+ disableDeletion: false
+ editable: false
+ options:
+ path: /var/lib/grafana/dashboards
diff --git
a/solr/monitoring/dev/grafana/provisioning/datasources/prometheus.yml
b/solr/monitoring/dev/grafana/provisioning/datasources/prometheus.yml
new file mode 100644
index 00000000000..1e877ed476e
--- /dev/null
+++ b/solr/monitoring/dev/grafana/provisioning/datasources/prometheus.yml
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Grafana datasource provisioning — automatically configures the Prometheus
+# datasource when Grafana starts (used by docker-compose.yml).
+
+apiVersion: 1
+
+datasources:
+ - name: Prometheus
+ type: prometheus
+ access: proxy
+ url: http://prometheus:9090
+ isDefault: true
+ editable: false
+ jsonData:
+ timeInterval: "15s"
diff --git a/solr/monitoring/dev/prometheus/prometheus-local.yml
b/solr/monitoring/dev/prometheus/prometheus-local.yml
new file mode 100644
index 00000000000..b3935124c3f
--- /dev/null
+++ b/solr/monitoring/dev/prometheus/prometheus-local.yml
@@ -0,0 +1,48 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# prometheus-local.yml — scrape config for a Solr you run yourself.
+#
+# Used by: ./stack.sh --local-solr
+#
+# 'host.docker.internal' resolves to the host machine from inside Docker (Mac,
+# Windows, and Linux with Docker 20.10+). docker-compose.yml adds the matching
+# extra_hosts entry so this also works on Linux.
+#
+# A single node on :8983 is scraped by default. Add further ports to the
targets
+# list to match a multi-node local SolrCloud setup.
+
+global:
+ scrape_interval: 15s
+ evaluation_interval: 15s
+
+rule_files:
+ - /etc/prometheus/rules/solr-alerts.yml
+
+# Hand firing alerts to the Alertmanager container in this stack; without this
+# the rules evaluate but nothing is ever delivered.
+alerting:
+ alertmanagers:
+ - static_configs:
+ - targets: ['alertmanager:9093']
+
+scrape_configs:
+ - job_name: solr
+ # The v1 endpoint, which every Solr 10.x build serves. Solr 10.1 and later
+ # also serve the shorter v2 alias /api/metrics.
+ metrics_path: /solr/admin/metrics
+ scrape_interval: 15s
+ static_configs:
+ - targets: ['host.docker.internal:8983']
diff --git a/solr/monitoring/dev/prometheus/prometheus.yml
b/solr/monitoring/dev/prometheus/prometheus.yml
new file mode 100644
index 00000000000..821ce876be7
--- /dev/null
+++ b/solr/monitoring/dev/prometheus/prometheus.yml
@@ -0,0 +1,43 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# prometheus.yml — scrape config for the bundled Solr nodes in the dev stack.
+#
+# Used by docker-compose.yml in this directory. The job name must match the
+# mixin's solrSelector (job="solr" by default), or the generated dashboard and
+# alert rules will not match any series.
+
+global:
+ scrape_interval: 15s
+ evaluation_interval: 15s
+
+rule_files:
+ - /etc/prometheus/rules/solr-alerts.yml
+
+# Hand firing alerts to the Alertmanager container in this stack; without this
+# the rules evaluate but nothing is ever delivered.
+alerting:
+ alertmanagers:
+ - static_configs:
+ - targets: ['alertmanager:9093']
+
+scrape_configs:
+ - job_name: solr
+ # The v1 endpoint, which every Solr 10.x build serves. Solr 10.1 and later
+ # also serve the shorter v2 alias /api/metrics.
+ metrics_path: /solr/admin/metrics
+ scrape_interval: 15s
+ static_configs:
+ - targets: ['solr1:8983', 'solr2:8983']
diff --git a/solr/monitoring/dev/stack.sh b/solr/monitoring/dev/stack.sh
new file mode 100755
index 00000000000..3d6129faef0
--- /dev/null
+++ b/solr/monitoring/dev/stack.sh
@@ -0,0 +1,130 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# stack.sh — convenience launcher for the Solr monitoring dev stack.
+#
+# Usage:
+# ./stack.sh Full stack: bundled Solr +
monitoring + trafficgen
+# ./stack.sh --local-solr Monitoring only; Prometheus scrapes
your local Solr
+# at localhost:8983 (no bundled Solr
started)
+# ./stack.sh --local-solr URL Same, but trafficgen is also pointed
at URL
+# (implies --trafficgen)
+# ./stack.sh --local-solr --trafficgen Local mode + trafficgen at default
localhost:8983
+# ./stack.sh --down Stop and remove all containers
+# ./stack.sh --help Show this help
+#
+# In --local-solr mode Prometheus uses prometheus/prometheus-local.yml which
scrapes
+# host.docker.internal:8983. Edit that file to add further nodes or change
the port.
+
+set -euo pipefail
+
+cd "$(dirname "$0")"
+
+LOCAL_MODE=false
+WITH_TRAFFICGEN=false
+CUSTOM_SOLR_URL=""
+DOWN_MODE=false
+
+usage() {
+ awk '/^# Usage:/,/^[^#]/' "$0" | grep '^#' | sed 's/^# \{0,1\}//'
+ exit 0
+}
+
+# Translate long options to short equivalents so getopts can process them
uniformly.
+_args=()
+for _a in "$@"; do
+ case "$_a" in
+ --local-solr) _args+=(-l) ;;
+ --trafficgen) _args+=(-t) ;;
+ --down) _args+=(-d) ;;
+ --help) _args+=(-h) ;;
+ *) _args+=("$_a") ;;
+ esac
+done
+set -- "${_args[@]+"${_args[@]}"}"
+
+while getopts "ltdh" opt; do
+ case "$opt" in
+ l) LOCAL_MODE=true ;;
+ t) WITH_TRAFFICGEN=true ;;
+ d) DOWN_MODE=true ;;
+ h) usage ;;
+ ?) echo "Unknown option. Run ./stack.sh --help for usage." >&2; exit 1 ;;
+ esac
+done
+shift $((OPTIND - 1))
+
+# Optional URL positional argument following -l / --local-solr
+if $LOCAL_MODE && [[ $# -gt 0 ]]; then
+ CUSTOM_SOLR_URL="$1"
+ WITH_TRAFFICGEN=true
+ shift
+fi
+
+if $DOWN_MODE; then
+ # Include the bundled-solr profile so solr1/solr2 are stopped even if they
were started
+ # with that profile active. Passing --profile to a non-running service is
harmless.
+ docker compose --profile bundled-solr down
+ echo "All containers stopped and removed."
+ exit 0
+fi
+
+if $LOCAL_MODE; then
+ export PROMETHEUS_CONFIG="./prometheus/prometheus-local.yml"
+
+ if [[ -n "$CUSTOM_SOLR_URL" ]]; then
+ # Replace 'localhost' with host.docker.internal so trafficgen can reach
the host
+ export SOLR_URL="${CUSTOM_SOLR_URL/localhost/host.docker.internal}"
+ else
+ export SOLR_URL="http://host.docker.internal:8983/solr";
+ fi
+
+ SERVICES="prometheus grafana alertmanager"
+ if $WITH_TRAFFICGEN; then
+ SERVICES="$SERVICES trafficgen"
+ fi
+
+ # shellcheck disable=SC2086
+ docker compose up -d $SERVICES
+
+ echo ""
+ echo "Monitoring stack started in local-Solr mode."
+ echo "Prometheus is scraping your local Solr via host.docker.internal"
+ if $WITH_TRAFFICGEN; then
+ echo "Traffic generator is running against: $SOLR_URL"
+ fi
+ echo ""
+ echo " Dashboard: http://localhost:3000/d/solr-mixin-example";
+ echo " Grafana: http://localhost:3000 (view without login; admin/admin
to edit)"
+ echo " Prometheus: http://localhost:9090";
+ echo " Alertmanager: http://localhost:9093";
+ echo ""
+ echo "Tip: edit prometheus/prometheus-local.yml to add further nodes or
change the port."
+else
+ # Bundled mode: activate the bundled-solr profile so solr1 and solr2 start
+ COMPOSE_PROFILES=bundled-solr docker compose up -d
+
+ echo ""
+ echo "Full monitoring stack started."
+ echo " Solr 1: http://localhost:8983";
+ echo " Solr 2: http://localhost:8984";
+ echo " Dashboard: http://localhost:3000/d/solr-mixin-example";
+ echo " Grafana: http://localhost:3000 (view without login;
admin/admin to edit)"
+ echo " Prometheus: http://localhost:9090";
+ echo " Alertmanager: http://localhost:9093";
+ echo ""
+ echo "Tip: Prometheus may show scrape errors for the first 30-60 seconds
while Solr starts."
+fi
diff --git a/solr/monitoring/dev/trafficgen/Dockerfile
b/solr/monitoring/dev/trafficgen/Dockerfile
new file mode 100644
index 00000000000..5fa4c237170
--- /dev/null
+++ b/solr/monitoring/dev/trafficgen/Dockerfile
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM python:3.12-slim
+
+WORKDIR /app
+
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+COPY trafficgen.py .
+COPY trafficgen.yml .
+
+RUN useradd -r -u 1000 -g root trafficgen
+USER 1000
+
+ENTRYPOINT ["python3", "trafficgen.py"]
diff --git a/solr/monitoring/dev/trafficgen/requirements.txt
b/solr/monitoring/dev/trafficgen/requirements.txt
new file mode 100644
index 00000000000..f84ec71246f
--- /dev/null
+++ b/solr/monitoring/dev/trafficgen/requirements.txt
@@ -0,0 +1,3 @@
+pysolr>=3.10.0
+requests>=2.31.0
+PyYAML>=6.0
diff --git a/solr/monitoring/dev/trafficgen/trafficgen.py
b/solr/monitoring/dev/trafficgen/trafficgen.py
new file mode 100644
index 00000000000..292092a789d
--- /dev/null
+++ b/solr/monitoring/dev/trafficgen/trafficgen.py
@@ -0,0 +1,298 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""trafficgen.py - Traffic generator for Apache Solr
+
+Generates continuous index and search traffic against a Solr endpoint.
+Configuration is read from a YAML file (default: trafficgen.yml in same
directory).
+If the config has no 'words' key, a vocabulary is generated in RAM at startup.
+
+Environment variable overrides:
+ STOP_AFTER=<secs> Stop after this many seconds (default: run
forever)
+ CONFIG_FILE=<path> Path to config file (default: trafficgen.yml)
+ INDEX_SLEEP_SECS=<secs> Override indexing.sleepSecs
+ SEARCH_SLEEP_SECS=<secs> Override searching.sleepSecs
+ DELETIONS_SLEEP_SECS=<secs> Override deletions.sleepSecs
+ QUIET=true Suppress per-operation log lines
+ SOLR_URL=<url> Override solrUrl
+"""
+
+import argparse
+import logging
+import os
+import random
+import sys
+import threading
+import time
+import uuid
+
+import pysolr
+import requests
+import yaml
+
+
+def generate_words(count=300):
+ """Generate a pronounceable in-RAM vocabulary of `count` unique words."""
+ consonants = "bcdfghjklmnprstvwz"
+ vowels = "aeiou"
+ words = set()
+ while len(words) < count:
+ length = random.randint(4, 9)
+ word = ""
+ for i in range(length):
+ word += random.choice(consonants if i % 2 == 0 else vowels)
+ words.add(word)
+ return sorted(words)
+
+
+def load_config(config_file):
+ with open(config_file, "r") as f:
+ return yaml.safe_load(f)
+
+
+def apply_env_overrides(config):
+ if os.environ.get("SOLR_URL"):
+ config["solrUrl"] = os.environ["SOLR_URL"]
+ if os.environ.get("INDEX_SLEEP_SECS"):
+ config["indexing"]["sleepSecs"] = float(os.environ["INDEX_SLEEP_SECS"])
+ # Support both correct and typo variant from spec
+ search_secs = os.environ.get("SEARCH_SLEEP_SECS") or
os.environ.get("SEARCH_SEEP_SECS")
+ if search_secs:
+ config["searching"]["sleepSecs"] = float(search_secs)
+ if os.environ.get("DELETIONS_SLEEP_SECS"):
+ config["deletions"]["sleepSecs"] =
float(os.environ["DELETIONS_SLEEP_SECS"])
+ return config
+
+
+def ensure_collection_exists(solr_url, collection_config):
+ name = collection_config["name"]
+ num_shards = collection_config.get("numShards", 1)
+ replication_factor = collection_config.get("replicationFactor", 1)
+ config_set = collection_config.get("configSet", "_default")
+
+ admin_url = f"{solr_url.rstrip('/')}/admin/collections"
+
+ try:
+ resp = requests.get(admin_url, params={"action": "LIST", "wt":
"json"}, timeout=10)
+ resp.raise_for_status()
+ collections = resp.json().get("collections", [])
+ except Exception as e:
+ logging.warning(f"CREATE: Could not list collections (non-cloud
Solr?): {e}")
+ return
+
+ if name in collections:
+ logging.info(f"CREATE: Collection '{name}' already exists")
+ return
+
+ logging.info(
+ f"CREATE: Creating collection '{name}' with numShards={num_shards}, "
+ f"replicationFactor={replication_factor}, configSet='{config_set}'"
+ )
+ try:
+ create_resp = requests.get(
+ admin_url,
+ params={
+ "action": "CREATE",
+ "name": name,
+ "numShards": num_shards,
+ "replicationFactor": replication_factor,
+ "collection.configName": config_set,
+ "wt": "json",
+ },
+ timeout=30,
+ )
+ create_resp.raise_for_status()
+ result = create_resp.json()
+ status = result.get("responseHeader", {}).get("status", -1)
+ if status != 0:
+ raise RuntimeError(f"Unexpected response status {status}:
{result}")
+ logging.info(f"CREATE: Collection '{name}' created successfully")
+ except Exception as e:
+ logging.error(f"CREATE: Failed to create collection '{name}': {e}")
+ sys.exit(1)
+
+
+def build_document(words, freetext_field, facet_field, num_words):
+ return {
+ "id": str(uuid.uuid4()),
+ freetext_field: " ".join(random.choices(words, k=num_words)),
+ facet_field: random.choice(words),
+ }
+
+
+def run_indexing(solr, config, words, stop_event, quiet):
+ batch_size = config["indexing"]["batchSize"]
+ sleep_secs = config["indexing"]["sleepSecs"]
+ freetext_field = config["indexing"]["freetextFieldName"]
+ facet_field = config["indexing"]["facetFieldName"]
+ num_words = config["indexing"]["numWords"]
+
+ while not stop_event.is_set():
+ docs = [build_document(words, freetext_field, facet_field, num_words)
for _ in range(batch_size)]
+ try:
+ t0 = time.monotonic()
+ solr.add(docs)
+ ms = int((time.monotonic() - t0) * 1000)
+ if not quiet:
+ logging.info(f"INDEX: batch={batch_size}
field={freetext_field} time={ms}ms")
+ except Exception as e:
+ logging.error(f"INDEX: {e}")
+ stop_event.wait(sleep_secs)
+
+
+def run_deletions(solr, config, words, stop_event, quiet):
+ sleep_secs = config["deletions"]["sleepSecs"]
+ freetext_field = config["indexing"]["freetextFieldName"]
+
+ while not stop_event.is_set():
+ word = random.choice(words)
+ q = f"{freetext_field}:{word}"
+ try:
+ t0 = time.monotonic()
+ solr.delete(q=q)
+ ms = int((time.monotonic() - t0) * 1000)
+ if not quiet:
+ logging.info(f"DELETE: q='{q}' time={ms}ms")
+ except Exception as e:
+ logging.error(f"DELETE: {e}")
+ stop_event.wait(sleep_secs)
+
+
+def run_searching(solr, config, words, stop_event, quiet):
+ sleep_secs = config["searching"]["sleepSecs"]
+ freetext_field = config["searching"]["freetextFieldName"]
+ facet_field = config["searching"]["facetFieldName"]
+ num_words = config["searching"]["numWords"]
+
+ while not stop_event.is_set():
+ query_words = random.choices(words, k=num_words)
+ if num_words == 1:
+ q = f"{freetext_field}:{query_words[0]}"
+ else:
+ q = f"{freetext_field}:({' '.join(query_words)})"
+ try:
+ t0 = time.monotonic()
+ results = solr.search(
+ q,
+ rows=10,
+ facet="true",
+ **{"facet.field": facet_field},
+ )
+ ms = int((time.monotonic() - t0) * 1000)
+ if not quiet:
+ logging.info(f"SEARCH: q='{q}' hits={results.hits}
time={ms}ms")
+ except Exception as e:
+ logging.error(f"SEARCH: {e}")
+ stop_event.wait(sleep_secs)
+
+
+def main():
+ parser = argparse.ArgumentParser(
+ description="Traffic generator for Apache Solr - continuously indexes
and searches documents.",
+ formatter_class=argparse.RawDescriptionHelpFormatter,
+ epilog=__doc__,
+ )
+ parser.parse_args()
+
+ logging.basicConfig(
+ level=logging.INFO,
+ format="%(asctime)s %(levelname)s %(message)s",
+ datefmt="%Y-%m-%dT%H:%M:%S",
+ stream=sys.stdout,
+ )
+ logging.getLogger("pysolr").setLevel(logging.WARNING)
+
+ config_file = os.environ.get("CONFIG_FILE")
+ if not config_file:
+ script_dir = os.path.dirname(os.path.abspath(__file__))
+ config_file = os.path.join(script_dir, "trafficgen.yml")
+
+ logging.info(f"Loading config from {config_file}")
+ config = load_config(config_file)
+ config = apply_env_overrides(config)
+
+ if config.get("words"):
+ words = config["words"]
+ logging.info(f"Vocabulary: {len(words)} words loaded from config")
+ else:
+ words = generate_words(300)
+ logging.info(f"Vocabulary: {len(words)} words auto-generated in RAM")
+
+ quiet = os.environ.get("QUIET", "").lower() == "true"
+ stop_after_secs = float(os.environ["STOP_AFTER"]) if
os.environ.get("STOP_AFTER") else None
+
+ solr_url = config["solrUrl"].rstrip("/")
+ collection_name = config["collection"]["name"]
+
+ logging.info(f"Solr URL: {solr_url}")
+ logging.info(f"Collection: {collection_name}")
+
+ ensure_collection_exists(solr_url, config["collection"])
+
+ solr = pysolr.Solr(
+ f"{solr_url}/{collection_name}",
+ always_commit=True,
+ timeout=10,
+ )
+
+ stop_event = threading.Event()
+
+ index_thread = threading.Thread(
+ target=run_indexing,
+ args=(solr, config, words, stop_event, quiet),
+ daemon=True,
+ name="indexer",
+ )
+ search_thread = threading.Thread(
+ target=run_searching,
+ args=(solr, config, words, stop_event, quiet),
+ daemon=True,
+ name="searcher",
+ )
+ deletion_thread = threading.Thread(
+ target=run_deletions,
+ args=(solr, config, words, stop_event, quiet),
+ daemon=True,
+ name="deletions",
+ )
+
+ index_thread.start()
+ search_thread.start()
+ deletion_thread.start()
+
+ if stop_after_secs:
+ logging.info(f"Traffic generation started, will stop after
{stop_after_secs}s")
+ else:
+ logging.info("Traffic generation started, running forever (set
STOP_AFTER to limit duration)")
+
+ try:
+ if stop_after_secs:
+ time.sleep(stop_after_secs)
+ logging.info("STOP_AFTER reached, shutting down")
+ else:
+ while True:
+ time.sleep(1)
+ except KeyboardInterrupt:
+ logging.info("Interrupted, shutting down")
+ finally:
+ stop_event.set()
+ index_thread.join(timeout=10)
+ search_thread.join(timeout=10)
+ deletion_thread.join(timeout=10)
+ logging.info("Traffic generation stopped")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/solr/monitoring/dev/trafficgen/trafficgen.yml
b/solr/monitoring/dev/trafficgen/trafficgen.yml
new file mode 100644
index 00000000000..58610b827c4
--- /dev/null
+++ b/solr/monitoring/dev/trafficgen/trafficgen.yml
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# trafficgen.yml — configuration for the dev docker-compose stack.
+#
+# Tuned for SolrCloud (numShards: 2) with faster indexing so the
+# Grafana dashboard panels show activity immediately.
+# Solr URL is overridden at runtime via SOLR_URL env var in docker-compose.yml.
+# Word list is auto-generated in RAM by trafficgen.py (no 'words' key needed).
+
+solrUrl: http://localhost:8983/solr
+
+collection:
+ name: test
+ numShards: 2
+ replicationFactor: 2
+ configSet: _default
+
+indexing:
+ batchSize: 10
+ sleepSecs: 5
+ freetextFieldName: text_t
+ numWords: 20
+ facetFieldName: keywords_s
+
+searching:
+ sleepSecs: 1
+ freetextFieldName: text_t
+ numWords: 1
+ facetFieldName: keywords_s
+
+deletions:
+ sleepSecs: 30
diff --git a/solr/monitoring/grafana-solr-dashboard.json
b/solr/monitoring/grafana-solr-dashboard.json
new file mode 100644
index 00000000000..e35648dc22e
--- /dev/null
+++ b/solr/monitoring/grafana-solr-dashboard.json
@@ -0,0 +1,149 @@
+{
+ "__license": "Licensed to the Apache Software Foundation (ASF) under one or
more contributor license agreements. See the NOTICE file distributed with this
work for additional information regarding copyright ownership. The ASF licenses
this file to You under the Apache License, Version 2.0 (the \"License\"); you
may not use this file except in compliance with the License. You may obtain a
copy of the License at http://www.apache.org/licenses/LICENSE-2.0 — Generated
by mixin/dashboards/ [...]
+ "description": "Placeholder dashboard that exercises the monitoring mixin
build. It is not the Solr dashboard — that is still being designed, see
SOLR-18534.",
+ "editable": true,
+ "panels": [
+ {
+ "datasource": {
+ "type": "datasource",
+ "uid": "-- Mixed --"
+ },
+ "description": "Requests per second per node, across all handlers and
cores.",
+ "fieldConfig": {
+ "defaults": {
+ "unit": "reqps"
+ }
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 0,
+ "y": 0
+ },
+ "id": 1,
+ "options": {
+ "legend": {
+ "displayMode": "list"
+ },
+ "tooltip": {
+ "mode": "multi"
+ }
+ },
+ "pluginVersion": "v11.4.0",
+ "targets": [
+ {
+ "datasource": {
+ "type": "prometheus",
+ "uid": "$datasource"
+ },
+ "expr": "sum by
(instance)(rate(solr_core_requests_total{job=\"solr\"}[$interval]))",
+ "legendFormat": "{{instance}}"
+ }
+ ],
+ "title": "Request Rate",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "datasource",
+ "uid": "-- Mixed --"
+ },
+ "description": "JVM heap in use per node, summed over the heap memory
pools. The inner max collapses pools reported by more than one OTel
instrumentation scope, which differ by Solr version.",
+ "fieldConfig": {
+ "defaults": {
+ "unit": "bytes"
+ }
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 12,
+ "y": 0
+ },
+ "id": 2,
+ "options": {
+ "legend": {
+ "displayMode": "list"
+ },
+ "tooltip": {
+ "mode": "multi"
+ }
+ },
+ "pluginVersion": "v11.4.0",
+ "targets": [
+ {
+ "datasource": {
+ "type": "prometheus",
+ "uid": "$datasource"
+ },
+ "expr": "sum by (instance)(max by (instance,
jvm_memory_pool_name)(jvm_memory_used_bytes{job=\"solr\",jvm_memory_type=\"heap\"}))",
+ "legendFormat": "{{instance}}"
+ }
+ ],
+ "title": "Heap Used",
+ "type": "timeseries"
+ }
+ ],
+ "refresh": "auto",
+ "schemaVersion": 36,
+ "tags": [
+ "solr",
+ "prometheus",
+ "example"
+ ],
+ "templating": {
+ "list": [
+ {
+ "label": "Data Source",
+ "name": "datasource",
+ "query": "prometheus",
+ "type": "datasource"
+ },
+ {
+ "current": {
+ "selected": false,
+ "text": "5m",
+ "value": "5m"
+ },
+ "label": "Interval",
+ "name": "interval",
+ "options": [
+ {
+ "selected": true,
+ "text": "1m",
+ "value": "1m"
+ },
+ {
+ "selected": false,
+ "text": "5m",
+ "value": "5m"
+ },
+ {
+ "selected": false,
+ "text": "10m",
+ "value": "10m"
+ },
+ {
+ "selected": false,
+ "text": "30m",
+ "value": "30m"
+ },
+ {
+ "selected": false,
+ "text": "1h",
+ "value": "1h"
+ }
+ ],
+ "query": "1m,5m,10m,30m,1h",
+ "type": "interval"
+ }
+ ]
+ },
+ "time": {
+ "from": "now-15m",
+ "to": "now"
+ },
+ "timezone": "browser",
+ "title": "Solr Mixin Example",
+ "uid": "solr-mixin-example"
+}
diff --git a/solr/monitoring/mixin/.gitignore b/solr/monitoring/mixin/.gitignore
new file mode 100644
index 00000000000..8cd29773f01
--- /dev/null
+++ b/solr/monitoring/mixin/.gitignore
@@ -0,0 +1,2 @@
+vendor/
+out/
diff --git a/solr/monitoring/mixin/Dockerfile b/solr/monitoring/mixin/Dockerfile
new file mode 100644
index 00000000000..6f01dd0a7be
--- /dev/null
+++ b/solr/monitoring/mixin/Dockerfile
@@ -0,0 +1,55 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# ============================================================================
+# Build image for the Solr monitoring mixin.
+# Contains: jsonnet, jsonnetfmt, jb (jsonnet-bundler), gojsontoyaml, promtool.
+#
+# Not intended for production use — developer tooling only.
+# See make.sh for usage.
+# ============================================================================
+
+# --- Stage 1: compile Go-based tools -----------------------------------------
+FROM golang:1-bookworm AS go-tools
+
+ENV CGO_ENABLED=0
+
+RUN go install github.com/google/go-jsonnet/cmd/[email protected] \
+ && go install github.com/google/go-jsonnet/cmd/[email protected] \
+ && go install github.com/jsonnet-bundler/jsonnet-bundler/cmd/[email protected] \
+ && go install github.com/brancz/[email protected]
+
+# --- Stage 2: extract promtool from the official Prometheus image
-------------
+FROM prom/prometheus:v2.52.0 AS prometheus
+
+# --- Stage 3: lean runtime image ---------------------------------------------
+FROM ubuntu:24.04
+
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends make ca-certificates \
+ && rm -rf /var/lib/apt/lists/*
+
+COPY --from=go-tools /go/bin/jsonnet /usr/local/bin/
+COPY --from=go-tools /go/bin/jsonnetfmt /usr/local/bin/
+COPY --from=go-tools /go/bin/jb /usr/local/bin/
+COPY --from=go-tools /go/bin/gojsontoyaml /usr/local/bin/
+COPY --from=prometheus /bin/promtool /usr/local/bin/
+
+# The mixin source tree is expected to be mounted at /work/mixin
+# (i.e., the monitoring/ directory is mounted as /work).
+WORKDIR /work/mixin
+
+ENTRYPOINT ["make"]
+CMD ["all"]
diff --git a/solr/monitoring/mixin/Makefile b/solr/monitoring/mixin/Makefile
new file mode 100644
index 00000000000..a764b38eaa7
--- /dev/null
+++ b/solr/monitoring/mixin/Makefile
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+JSONNET = jsonnet
+JSONNETFMT = jsonnetfmt
+JB = jb
+GOJSONTOYAML = gojsontoyaml
+PROMTOOL = promtool
+
+JSONNET_FLAGS = -J vendor
+FMT_FLAGS = -n 2 --max-blank-lines 2 --string-style s --comment-style s
+
+DASHBOARD_OUT = ../grafana-solr-dashboard.json
+ALERTS_OUT = ../prometheus-solr-alerts.yml
+
+# Generated artifacts are written to a temp file and moved into place only on
+# success, so a missing tool or a jsonnet error can never leave a committed
+# artifact truncated. pipefail makes a failure anywhere in a pipe fatal.
+SHELL := /bin/bash
+.SHELLFLAGS := -eu -o pipefail -c
+
+# Abort before touching any output if a required tool is missing.
+require_tool = @command -v $(1) >/dev/null 2>&1 || { \
+ echo "ERROR: required tool '$(1)' not found in PATH." >&2; \
+ echo " Install the local toolchain (see ../README.md, 'Mixin
Build')," >&2; \
+ echo " or run ./make.sh $@ to build inside Docker instead."
>&2; \
+ exit 1; }
+
+require_vendor = @test -d vendor || { \
+ echo "ERROR: vendor/ is missing — run 'make install' (or ./make.sh
install) first." >&2; \
+ exit 1; }
+
+.PHONY: all dashboards alerts install check fmt clean
+
+all: dashboards alerts
+
+install:
+ $(call require_tool,$(JB))
+ $(JB) install
+ @echo "→ Installed jsonnet dependencies into vendor/"
+
+dashboards:
+ $(call require_tool,$(JSONNET))
+ $(require_vendor)
+ $(JSONNET) $(JSONNET_FLAGS) \
+ -e '(import
"mixin.libsonnet").grafanaDashboards["grafana-solr-dashboard.json"]' \
+ > $(DASHBOARD_OUT).tmp || { rm -f $(DASHBOARD_OUT).tmp; exit 1; }
+ mv $(DASHBOARD_OUT).tmp $(DASHBOARD_OUT)
+ @echo "→ Generated $(DASHBOARD_OUT)"
+
+alerts:
+ $(call require_tool,$(JSONNET))
+ $(call require_tool,$(GOJSONTOYAML))
+ $(require_vendor)
+ cat license-header.yml > $(ALERTS_OUT).tmp
+ $(JSONNET) $(JSONNET_FLAGS) \
+ -e '(import "mixin.libsonnet").prometheusAlerts' \
+ | $(GOJSONTOYAML) >> $(ALERTS_OUT).tmp || { rm -f $(ALERTS_OUT).tmp;
exit 1; }
+ mv $(ALERTS_OUT).tmp $(ALERTS_OUT)
+ @echo "→ Generated $(ALERTS_OUT)"
+
+check:
+ $(call require_tool,$(PROMTOOL))
+ $(PROMTOOL) check rules $(ALERTS_OUT)
+ @echo "→ Alert rules validated OK"
+
+fmt:
+ $(call require_tool,$(JSONNETFMT))
+ find . -path ./vendor -prune -o -name '*.libsonnet' -exec $(JSONNETFMT)
$(FMT_FLAGS) -i {} \;
+ @echo "→ Formatted all .libsonnet files (vendor/ excluded)"
+
+clean:
+ rm -f $(DASHBOARD_OUT) $(ALERTS_OUT) $(DASHBOARD_OUT).tmp
$(ALERTS_OUT).tmp
+ @echo "→ Removed generated artifacts"
diff --git a/solr/monitoring/mixin/alerts/alerts.libsonnet
b/solr/monitoring/mixin/alerts/alerts.libsonnet
new file mode 100644
index 00000000000..93dad7f5fa2
--- /dev/null
+++ b/solr/monitoring/mixin/alerts/alerts.libsonnet
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// alerts.libsonnet — placeholder alert rule.
+//
+// One rule, enough to show that the mixin build produces a file promtool
+// accepts and Prometheus loads. The real rule set replaces this wholesale;
+// see SOLR-18535.
+//
+// The rules are built from the configuration of the merged mixin root
+// ($._config), not from a direct import, so that callers can override settings
+// without editing this tree.
+
+local alertsFor(cfg) = {
+ groups: [
+ {
+ name: 'SolrExampleAlerts',
+ rules: [
+ {
+ alert: 'SolrNodeDown',
+ expr: 'up{%s} == 0' % cfg.solrSelector,
+ 'for': '2m',
+ labels: { severity: 'warning' },
+ annotations: {
+ summary: 'Solr node {{ $labels.instance }} is not responding to
scrapes',
+ description: |||
+ Prometheus has failed to scrape {{ $labels.instance }} for 2
minutes.
+ Either the node is down or its metrics endpoint is unreachable.
+ |||,
+ },
+ },
+ ],
+ },
+ ],
+};
+
+{
+ prometheusAlerts+:: alertsFor($._config),
+}
diff --git a/solr/monitoring/mixin/config.libsonnet
b/solr/monitoring/mixin/config.libsonnet
new file mode 100644
index 00000000000..342948d14dd
--- /dev/null
+++ b/solr/monitoring/mixin/config.libsonnet
@@ -0,0 +1,35 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// config.libsonnet — settings you can override without editing the dashboard
+// or alert sources.
+//
+// Every key here is consumed by dashboards/ or alerts/; nothing is declared
+// ahead of its use. To override, import the mixin and patch _config:
+//
+// (import 'mixin.libsonnet') + { _config+:: { solrSelector: 'job="my-solr"'
} }
+
+{
+ _config:: {
+ // Label selector that identifies Solr targets in your Prometheus. Applied
to
+ // every dashboard query and alert expression, so JVM metrics scraped from
+ // other Java services are not mistaken for Solr's.
+ solrSelector: 'job="solr"',
+
+ // Window for rate() queries, and the preselected value of the dashboard's
+ // Interval dropdown. Raise it for sparse scrape intervals.
+ rateInterval: '5m',
+ },
+}
diff --git a/solr/monitoring/mixin/dashboards/dashboards.libsonnet
b/solr/monitoring/mixin/dashboards/dashboards.libsonnet
new file mode 100644
index 00000000000..d214c72dbd7
--- /dev/null
+++ b/solr/monitoring/mixin/dashboards/dashboards.libsonnet
@@ -0,0 +1,102 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// dashboards.libsonnet — placeholder dashboard.
+//
+// Two panels, enough to show that the mixin build produces a dashboard Grafana
+// can load and that config.libsonnet overrides reach the generated queries.
+// The real dashboard replaces this wholesale; see SOLR-18534.
+//
+// The dashboard is built from the configuration of the merged mixin root
+// ($._config), not from a direct import, so that callers can override settings
+// without editing this tree.
+
+local g = import
'github.com/grafana/grafonnet/gen/grafonnet-latest/main.libsonnet';
+
+local d = g.dashboard;
+local p = g.panel;
+local q = g.query.prometheus;
+local v = g.dashboard.variable;
+
+local licenseComment =
+ 'Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. '
+ + 'See the NOTICE file distributed with this work for additional information
regarding copyright ownership. '
+ + 'The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); '
+ + 'you may not use this file except in compliance with the License. '
+ + 'You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0 '
+ + '— Generated by mixin/dashboards/dashboards.libsonnet. To regenerate: cd
mixin && make dashboards';
+
+local prom(expr, legend='{{instance}}') =
+ q.new('$datasource', expr)
+ + q.withLegendFormat(legend);
+
+local ts(title, exprs, unit='short', desc='') =
+ p.timeSeries.new(title)
+ + p.timeSeries.queryOptions.withTargets(exprs)
+ + p.timeSeries.standardOptions.withUnit(unit)
+ + p.timeSeries.panelOptions.withDescription(desc)
+ + p.timeSeries.options.legend.withDisplayMode('list')
+ + p.timeSeries.options.tooltip.withMode('multi');
+
+local dashboardFor(cfg) =
+ local datasourceVar =
+ v.datasource.new('datasource', 'prometheus')
+ + v.datasource.generalOptions.withLabel('Data Source');
+
+ local intervalVar =
+ v.interval.new('interval', ['1m', '5m', '10m', '30m', '1h'])
+ + v.interval.generalOptions.withCurrent(cfg.rateInterval)
+ + v.interval.generalOptions.withLabel('Interval');
+
+ local panels = [
+ ts(
+ 'Request Rate',
+ [prom('sum by (instance)(rate(solr_core_requests_total{%s}[$interval]))'
% cfg.solrSelector)],
+ unit='reqps',
+ desc='Requests per second per node, across all handlers and cores.'
+ ) + { gridPos: { x: 0, y: 0, w: 12, h: 8 } },
+
+ ts(
+ 'Heap Used',
+ [prom(
+ 'sum by (instance)(max by (instance,
jvm_memory_pool_name)(jvm_memory_used_bytes{%s,jvm_memory_type="heap"}))' %
cfg.solrSelector
+ )],
+ unit='bytes',
+ desc='JVM heap in use per node, summed over the heap memory pools. The
inner max collapses pools reported by more than one OTel instrumentation scope,
which differ by Solr version.'
+ ) + { gridPos: { x: 12, y: 0, w: 12, h: 8 } },
+ ];
+
+ { __license: licenseComment }
+ + d.new('Solr Mixin Example')
+ + d.withUid('solr-mixin-example')
+ + d.withDescription(
+ 'Placeholder dashboard that exercises the monitoring mixin build. '
+ + 'It is not the Solr dashboard — that is still being designed, see
SOLR-18534.'
+ )
+ + d.withTags(['solr', 'prometheus', 'example'])
+ + d.withRefresh('auto')
+ + d.withTimezone('browser')
+ + d.withEditable(true)
+ + d.withSchemaVersion(36)
+ + d.time.withFrom('now-15m')
+ + d.time.withTo('now')
+ + d.withVariables([datasourceVar, intervalVar])
+ + d.withPanels(panels);
+
+{
+ grafanaDashboards+:: {
+ 'grafana-solr-dashboard.json': dashboardFor($._config),
+ },
+}
diff --git a/solr/monitoring/mixin/jsonnetfile.json
b/solr/monitoring/mixin/jsonnetfile.json
new file mode 100644
index 00000000000..2d56d912452
--- /dev/null
+++ b/solr/monitoring/mixin/jsonnetfile.json
@@ -0,0 +1,15 @@
+{
+ "version": 1,
+ "dependencies": [
+ {
+ "source": {
+ "git": {
+ "remote": "https://github.com/grafana/grafonnet.git";,
+ "subdir": "gen/grafonnet-latest"
+ }
+ },
+ "version": "main"
+ }
+ ],
+ "legacyImports": false
+}
diff --git a/solr/monitoring/mixin/jsonnetfile.lock.json
b/solr/monitoring/mixin/jsonnetfile.lock.json
new file mode 100644
index 00000000000..190410bcefd
--- /dev/null
+++ b/solr/monitoring/mixin/jsonnetfile.lock.json
@@ -0,0 +1,46 @@
+{
+ "version": 1,
+ "dependencies": [
+ {
+ "source": {
+ "git": {
+ "remote": "https://github.com/grafana/grafonnet.git";,
+ "subdir": "gen/grafonnet-latest"
+ }
+ },
+ "version": "7380c9c64fb973f34c3ec46265621a2b0dee0058",
+ "sum": "V9vAj21qJOc2DlMPDgB1eEjSQU4A+sAA4AXuJ6bd4xc="
+ },
+ {
+ "source": {
+ "git": {
+ "remote": "https://github.com/grafana/grafonnet.git";,
+ "subdir": "gen/grafonnet-v11.4.0"
+ }
+ },
+ "version": "7380c9c64fb973f34c3ec46265621a2b0dee0058",
+ "sum": "aVAX09paQYNOoCSKVpuk1exVIyBoMt/C50QJI+Q/3nA="
+ },
+ {
+ "source": {
+ "git": {
+ "remote": "https://github.com/jsonnet-libs/docsonnet.git";,
+ "subdir": "doc-util"
+ }
+ },
+ "version": "6ac6c69685b8c29c54515448eaca583da2d88150",
+ "sum": "BrAL/k23jq+xy9oA7TWIhUx07dsA/QLm3g7ktCwe//U="
+ },
+ {
+ "source": {
+ "git": {
+ "remote": "https://github.com/jsonnet-libs/xtd.git";,
+ "subdir": ""
+ }
+ },
+ "version": "4d7f8cb24d613430799f9d56809cc6964f35cea9",
+ "sum": "hOrwkOx34tOXqoDVnwuI/Uf/dr9HFFSPWpDPOvnEGrk="
+ }
+ ],
+ "legacyImports": false
+}
diff --git a/solr/monitoring/mixin/license-header.yml
b/solr/monitoring/mixin/license-header.yml
new file mode 100644
index 00000000000..c8d2335f282
--- /dev/null
+++ b/solr/monitoring/mixin/license-header.yml
@@ -0,0 +1,17 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# This file is generated by mixin/alerts/alerts.libsonnet — do not edit
directly.
+# To regenerate: cd mixin && make alerts
diff --git a/solr/monitoring/mixin/make.sh b/solr/monitoring/mixin/make.sh
new file mode 100755
index 00000000000..7392e564da7
--- /dev/null
+++ b/solr/monitoring/mixin/make.sh
@@ -0,0 +1,67 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# make.sh — Docker-based wrapper around the mixin Makefile.
+#
+# Usage:
+# ./make.sh # equivalent to: make all
+# ./make.sh install all # equivalent to: make install all
+# ./make.sh check # equivalent to: make check
+# ./make.sh --rebuild # force-rebuild the Docker image, then make all
+#
+# The Docker image (solr-mixin-make:latest) is built automatically on first
+# use and reused on subsequent runs. Pass --rebuild to force a fresh build
+# (e.g. after updating the Dockerfile or tool versions).
+
+set -euo pipefail
+
+IMAGE="solr-mixin-make:latest"
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+MONITORING_DIR="$(dirname "$SCRIPT_DIR")"
+
+# --- parse flags -------------------------------------------------------------
+REBUILD=false
+MAKE_ARGS=()
+for arg in "$@"; do
+ case "$arg" in
+ --rebuild) REBUILD=true ;;
+ *) MAKE_ARGS+=("$arg") ;;
+ esac
+done
+
+# --- build image if missing or explicitly requested --------------------------
+if ! docker image inspect "$IMAGE" >/dev/null 2>&1 || [ "$REBUILD" = true ];
then
+ echo "[make.sh] Building Docker image $IMAGE ..."
+ docker build -t "$IMAGE" "$SCRIPT_DIR"
+fi
+
+# --- run make inside the container -------------------------------------------
+# Mount the monitoring/ directory as /work so that the Makefile's relative
+# paths (../grafana-solr-dashboard.json, ../prometheus-solr-alerts.yml) resolve
+# correctly to the host filesystem.
+TTY_FLAG=""
+[ -t 0 ] && TTY_FLAG="-t"
+
+# Run as the invoking user, so that the regenerated artifacts and vendor/ end
up
+# owned by you rather than by root. Without this, a build on native Linux
leaves
+# files you cannot edit or clean without sudo. The tools are static binaries
and
+# do not need the user to exist in the image's passwd database, but they do
need
+# a writable HOME.
+docker run --rm -i $TTY_FLAG \
+ --user "$(id -u):$(id -g)" \
+ -e HOME=/tmp \
+ -v "$MONITORING_DIR:/work" \
+ "$IMAGE" "${MAKE_ARGS[@]:-all}"
diff --git a/solr/monitoring/mixin/mixin.libsonnet
b/solr/monitoring/mixin/mixin.libsonnet
new file mode 100644
index 00000000000..13902679e7d
--- /dev/null
+++ b/solr/monitoring/mixin/mixin.libsonnet
@@ -0,0 +1,36 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// mixin.libsonnet — root entry point for the Solr monitoring mixin.
+//
+// Composes the configuration with the dashboards and alerts, so that both read
+// their settings from the merged root. That late binding is what lets a caller
+// override settings without editing this tree:
+//
+// (import 'mixin.libsonnet') + { _config+:: { solrSelector: 'job="my-solr"'
} }
+//
+// Exposes:
+// grafanaDashboards:: — map of filename -> Grafana dashboard JSON object
+// prometheusAlerts:: — Prometheus alert groups object
+//
+// Usage:
+// jsonnet -J vendor -e '(import
"mixin.libsonnet").grafanaDashboards["grafana-solr-dashboard.json"]' \
+// > ../grafana-solr-dashboard.json
+// jsonnet -J vendor -e '(import "mixin.libsonnet").prometheusAlerts' \
+// | gojsontoyaml > ../prometheus-solr-alerts.yml
+
+(import 'config.libsonnet')
++ (import 'dashboards/dashboards.libsonnet')
++ (import 'alerts/alerts.libsonnet')
diff --git a/solr/monitoring/prometheus-solr-alerts.yml
b/solr/monitoring/prometheus-solr-alerts.yml
new file mode 100644
index 00000000000..d1cefdcba1a
--- /dev/null
+++ b/solr/monitoring/prometheus-solr-alerts.yml
@@ -0,0 +1,30 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# This file is generated by mixin/alerts/alerts.libsonnet — do not edit
directly.
+# To regenerate: cd mixin && make alerts
+groups:
+- name: SolrExampleAlerts
+ rules:
+ - alert: SolrNodeDown
+ annotations:
+ description: |
+ Prometheus has failed to scrape {{ $labels.instance }} for 2 minutes.
+ Either the node is down or its metrics endpoint is unreachable.
+ summary: Solr node {{ $labels.instance }} is not responding to scrapes
+ expr: up{job="solr"} == 0
+ for: 2m
+ labels:
+ severity: warning
diff --git a/solr/solr-ref-guide/modules/deployment-guide/deployment-nav.adoc
b/solr/solr-ref-guide/modules/deployment-guide/deployment-nav.adoc
index e22a51c7be1..da291cbf746 100644
--- a/solr/solr-ref-guide/modules/deployment-guide/deployment-nav.adoc
+++ b/solr/solr-ref-guide/modules/deployment-guide/deployment-nav.adoc
@@ -105,6 +105,7 @@
** xref:configuring-logging.adoc[]
** xref:opentelemetry.adoc[]
** xref:metrics-reporting.adoc[]
+** xref:monitoring-with-prometheus-and-grafana.adoc[]
** xref:distributed-tracing.adoc[]
** xref:performance-statistics-reference.adoc[]
** xref:jmx-with-solr.adoc[]
diff --git
a/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/grafana-solr-dashboard.png
b/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/grafana-solr-dashboard.png
index 69f21a4debb..bc7b6ef2841 100644
Binary files
a/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/grafana-solr-dashboard.png
and
b/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/grafana-solr-dashboard.png
differ
diff --git
a/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/prometheus-solr-ping.png
b/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/prometheus-solr-ping.png
deleted file mode 100644
index e4bd7f77cfe..00000000000
Binary files
a/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/prometheus-solr-ping.png
and /dev/null differ
diff --git
a/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/solr-exporter-diagram.png
b/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/solr-exporter-diagram.png
deleted file mode 100644
index 96efece464d..00000000000
Binary files
a/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/solr-exporter-diagram.png
and /dev/null differ
diff --git
a/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/solr-monitoring-diagram.png
b/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/solr-monitoring-diagram.png
new file mode 100644
index 00000000000..dce7b383b46
Binary files /dev/null and
b/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/solr-monitoring-diagram.png
differ
diff --git
a/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/solr-monitoring-diagram.svg
b/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/solr-monitoring-diagram.svg
new file mode 100644
index 00000000000..bd01251f8e9
Binary files /dev/null and
b/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/solr-monitoring-diagram.svg
differ
diff --git
a/solr/solr-ref-guide/modules/deployment-guide/pages/metrics-reporting.adoc
b/solr/solr-ref-guide/modules/deployment-guide/pages/metrics-reporting.adoc
index 880359491fd..d47dd9ce630 100644
--- a/solr/solr-ref-guide/modules/deployment-guide/pages/metrics-reporting.adoc
+++ b/solr/solr-ref-guide/modules/deployment-guide/pages/metrics-reporting.adoc
@@ -180,6 +180,8 @@ See
https://prometheus.io/docs/concepts/data_model/[Prometheus Data Model] docum
This endpoint can be used to pull/scrape metrics to a Prometheus server or any
Prometheus-compatible backend directly from Solr.
+TIP: For a worked Prometheus and Grafana setup, including the dashboard and
alert rules Solr ships, see xref:monitoring-with-prometheus-and-grafana.adoc[].
+
==== Prometheus Setup
The `prometheus-config.yml` file needs to be configured for a Prometheus
server to scrape and collect metrics. A basic configuration for SolrCloud mode
is as follows:
diff --git
a/solr/solr-ref-guide/modules/deployment-guide/pages/monitoring-with-prometheus-and-grafana.adoc
b/solr/solr-ref-guide/modules/deployment-guide/pages/monitoring-with-prometheus-and-grafana.adoc
new file mode 100644
index 00000000000..3d2d720402b
--- /dev/null
+++
b/solr/solr-ref-guide/modules/deployment-guide/pages/monitoring-with-prometheus-and-grafana.adoc
@@ -0,0 +1,171 @@
+= Monitoring Solr with Prometheus and Grafana
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+[IMPORTANT]
+====
+*Work in progress — the artifacts shipped today are placeholders.*
+
+Solr 10 removed the standalone `solr-exporter` (the `prometheus-exporter`
module) and now exposes metrics natively, under OpenTelemetry-based metric
names.
+Dashboards and alert rules written against the old exporter therefore do not
work on Solr 10 or later: the process they scraped is gone, and the metric
names and labels they query have changed.
+Solr's own dashboard and alert rules are consequently being built from
scratch, panel by panel and rule by rule.
+
+What is in `solr/monitoring/` right now is a two-panel dashboard and a single
alert, present to prove the generation pipeline works — not a monitoring setup
to deploy.
+The dashboard worth deploying arrives in
https://issues.apache.org/jira/browse/SOLR-18534[SOLR-18534], the alert rules
in https://issues.apache.org/jira/browse/SOLR-18535[SOLR-18535], and the OTLP
Collector guidance in
https://issues.apache.org/jira/browse/SOLR-18536[SOLR-18536].
+Expect both files, and this page, to change substantially before they are
useful.
+====
+
+Solr ships two monitoring artifacts in `solr/monitoring/`:
+
+* `grafana-solr-dashboard.json` — a Grafana dashboard
+* `prometheus-solr-alerts.yml` — a set of Prometheus alert rules
+
+Import these into your own Prometheus + Grafana installation.
+Solr exposes all metrics natively, so no exporter process is required.
+For the full list of available metrics see xref:metrics-reporting.adoc[].
+
+.Solr monitoring architecture — Prometheus pull path and OTLP push path
+image::monitoring-with-prometheus-and-grafana/solr-monitoring-diagram.png[Solr
monitoring architecture diagram,800]
+
+== The Grafana Dashboard
+
+Import the dashboard by going to *Dashboards → Import* in Grafana and
uploading `grafana-solr-dashboard.json`.
+
+.The placeholder dashboard, running against two Solr nodes
+image::monitoring-with-prometheus-and-grafana/grafana-solr-dashboard.png[Grafana
dashboard screenshot,800]
+
+Dropdowns at the top of the dashboard scope what the panels show.
+The placeholder has two: `datasource`, so one import can serve several
Prometheus instances, and `interval`, which sets the window for rate
calculations.
+
+[[metrics-collection]]
+== Getting Metrics into Prometheus
+
+There are two ways to get Solr metrics into Prometheus.
+
+[[prometheus-scrape]]
+=== Option 1: Prometheus Pull (recommended)
+
+Prometheus scrapes each Solr node directly — no extra process needed.
+Add a scrape job to your `prometheus.yml`:
+
+[source,yaml]
+----
+scrape_configs:
+ - job_name: solr # <1>
+ metrics_path: /solr/admin/metrics # <2>
+ scrape_interval: 15s
+ static_configs:
+ - targets: ['solr-node1:8983', 'solr-node2:8983']
+----
+<1> The dashboard and alert rules select on this job name.
+ If you use another, set `solrSelector` to match — see <<customizing>>.
+<2> Solr 10.1 and later also serve this at the shorter v2 alias `/api/metrics`.
+
+Scrape every node rather than one: each reports its own JVM and the cores it
hosts.
+If metrics collection is switched off for a node, both endpoints answer `HTTP
510` and Prometheus marks the target down.
+
+[[otlp-push]]
+=== Option 2: OTLP Push via OpenTelemetry Collector
+
+When Solr's xref:opentelemetry.adoc[OpenTelemetry module] is enabled, Solr can
push metrics (and traces) via OTLP to an
https://opentelemetry.io/docs/collector/[OpenTelemetry Collector], which can
forward them to Prometheus or to any other OTLP-compatible backend.
+
+NOTE: An example Collector configuration, and the labelling needed to make
both paths produce identical series, will be documented here as part of
https://issues.apache.org/jira/browse/SOLR-18536[SOLR-18536].
+
+[[alert-rules]]
+== Alert Rules
+
+Reference `prometheus-solr-alerts.yml` from your `prometheus.yml`:
+
+[source,yaml]
+----
+rule_files:
+ - /path/to/prometheus-solr-alerts.yml
+----
+
+Validate the file before deploying it, and after any edit:
+
+[source,bash]
+----
+promtool check rules prometheus-solr-alerts.yml
+----
+
+Rules carry a `severity` label for routing in Alertmanager.
+One placeholder rule is included today:
+
+[cols="2,1,3",options="header"]
+|===
+| Alert | Severity | Fires when
+| `SolrNodeDown` | warning | A Solr target has failed to scrape for 2 minutes
+|===
+
+NOTE: The rules covering heap, GC, disk, latency, error rate and SolrCloud
health arrive in https://issues.apache.org/jira/browse/SOLR-18535[SOLR-18535].
+
+[[customizing]]
+== Customizing the Artifacts
+
+Both artifacts are generated from a https://jsonnet.org/[jsonnet] mixin source
in `solr/monitoring/mixin/`, following the conventions of the
https://github.com/monitoring-mixins/docs[monitoring-mixins] community.
+Edit the jsonnet, not the generated output.
+
+Tunable settings are centralized in `mixin/config.libsonnet`:
+
+[source,jsonnet]
+----
+{
+ _config:: {
+ solrSelector: 'job="solr"', // label selector identifying Solr targets
+ rateInterval: '5m', // window for rate() queries
+ },
+}
+----
+
+`solrSelector` is applied to every dashboard query and alert expression, so
JVM metrics scraped from other Java services are not read as Solr's.
+Either edit the file and regenerate, or leave this tree untouched and patch
the configuration from your own jsonnet:
+
+[source,jsonnet]
+----
+(import 'mixin.libsonnet') + { _config+:: { solrSelector: 'job="my-solr"' } }
+----
+
+Regenerating requires `jsonnet`, `jsonnet-bundler` and `gojsontoyaml`, or the
included Docker wrapper that supplies all three:
+
+[source,bash]
+----
+cd solr/monitoring/mixin
+./make.sh install # fetch jsonnet dependencies
+./make.sh all # regenerate both artifacts
+----
+
+See `solr/monitoring/README.md` for the remaining build targets and the local
toolchain alternative.
+
+[[dev-stack]]
+== Trying It Locally
+
+The Solr source tree carries a docker-compose stack that runs Solr,
Prometheus, Grafana and Alertmanager together with the generated artifacts
already wired in.
+It exists to make changes to those artifacts easy to see, and is contributor
tooling rather than a deployment to model yours on — it is not part of the Solr
release.
+
+[source,bash]
+----
+cd solr/monitoring/dev
+./stack.sh
+----
+
+Grafana is then on `http://localhost:3000`, opening directly on the
provisioned dashboard, with Prometheus on `:9090` and Alertmanager on `:9093`.
+`./stack.sh --local-solr` skips the bundled Solr nodes and scrapes a Solr you
are already running.
+
+[[distributed-tracing]]
+== Metrics and Tracing Together
+
+The same xref:opentelemetry.adoc[OpenTelemetry] instrumentation that produces
these metrics also emits xref:distributed-tracing.adoc[distributed traces], so
a latency spike on the dashboard can be followed into the traces of the
requests that caused it.
+Both signals are exported via OTLP and can flow through the same Collector
pipeline to a unified observability backend, or independently to specialized
ones.
