janhoy opened a new issue, #98: URL: https://github.com/apache/solr-orbit/issues/98
### Is your feature request related to a problem? Please describe Today a benchmark is one invocation of `solr-orbit run` against one Solr version, and the report lands in `~/.solr-orbit/benchmarks/test-runs/<id>/test_run.json`. That is fine for a human answering a question once. It does not let us answer "did Solr get slower between 9.10 and 9.11, and when exactly?", because that needs the same workload run reproducibly across many versions, a record of *where and how* each run executed, and each result placed on a timeline derived from Solr's git history rather than from whenever we happened to run it. ## End goal A matrix of Solr versions × workloads × hardware profiles × cluster configs, executed unattended from CI, producing one immutable enriched JSON report per cell in pluggable storage (local filesystem for development, a cloud bucket in production). The website that visualizes those reports is a separate tool and a separate issue. ### Describe the solution you'd like `launchr` is a Python CLI bundled in this repo. For each cell of the matrix, grouped by hardware profile so infrastructure is provisioned once and reused, it provisions infrastructure, deploys the Solr version under test, health-checks it and reads `/solr/admin/info/system` to capture the exact commit and JVM that actually ran, invokes `solr-orbit run --pipeline=benchmark-only`, enriches the resulting `test_run.json` with that metadata, uploads it, and tears down. A failed cell produces a `"status": "failed"` report and does not abort the rest of the matrix; infrastructure is torn down regardless of outcome. Provisioning strategies, Solr install methods, and storage backends are plugins behind `Protocol` interfaces, so the first increment can be laptop-only (local Docker, local filesystem, no new pip dependencies) and EC2, EKS, and S3 can arrive later without reshaping anything. Enrichment is additive: Solr Orbit's keys are written verbatim and launchr's metadata goes in a separate `launchr` block, so `solr-orbit compare` and `TestRun.from_dict()` keep working on stored reports. ### Describe alternatives you've considered Embedding it in the main solr-orbit tool Doing it as terraform in a different repo ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
