janhoy opened a new issue, #131: URL: https://github.com/apache/solr-orbit/issues/131
Two tools, built in phases, so that Solr's performance over time is measured on consistent hardware and published as a trend site. This issue is the roadmap: what the phases are, what each one unlocks, and where each stands. The tools themselves are tracked in #98 (launchr) and #99 (reportr). **The shape of it.** `solr-orbit` measures one cluster once. `launchr` walks a matrix of Solr version × workload × hardware profile × cluster config, owning the Solr lifecycle around each cell and storing one enriched report per cell. `reportr` turns those reports into a static site. The x-axis is the version timeline, not the run date, because the question is how Solr changed rather than when somebody happened to benchmark it. **Phases** | | Phase | Status | |---|---|---| | 1 | launchr: local MVP — matrix, local Docker, filesystem results | #108, in review | | 2 | reportr: local MVP — static trends site from stored reports | #109, in review | | 3 | S3 storage — the same contract as local, so switching is one config line | #130, in review | | 4 | Dedicated hardware — EC2 provisioning, SSH deploy, teardown | #130, in review | | 5 | Documentation — continuous-benchmarking guide and CLI references | #130, in review | | 6 | Kubernetes via kind, with the provisioning plugins generalised | not started | | 7 | Managed Kubernetes — EKS and AKS as thin provisioners over phase 6 | not started | | 8 | Scheduled runs from CI | not started | | 9 | reportr in production — publishing to an ASF-hosted location | not started, blocked on #110 | Phases 3, 4 and 5 landed together because dedicated hardware is only useful once there is somewhere durable to put what it produces, and neither is reviewable without documentation. **What is deliberately not here yet.** No automatic regression detection: the site shows what was measured and labels what the numbers cannot say, leaving the judgement to a reader. Multi-iteration runs with aggregation are refused rather than silently mishandled, because Solr Orbit generates its own ids for iterations after the first and launchr cannot locate those reports — supporting it needs launchr to drive the iterations itself. **Open questions that the phases do not answer.** - Which workloads are the publishable set, and what is each one's runtime budget? This bounds a nightly matrix against both a time ceiling and a bill. - Where does a published site live, and how does content reach it? #110. - Who owns the cloud account and the budget? There is no project account today; @janhoy is sponsoring the AWS usage personally until #110 settles where this runs and who pays for it. - Does run-to-run variance on identical hardware stay below the regressions worth catching? If not, a trend site is decorative. Measuring that band is the first thing dedicated hardware makes possible. 🤖 Assisted by Claude Code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
