janhoy opened a new pull request, #130: URL: https://github.com/apache/solr-orbit/pull/130
Adds an S3 storage backend and EC2 provisioning, so a matrix can run on AWS hardware nobody else is using and the results outlive it. Both are registered plugins — a storage backend and a provisioning strategy — so an Azure or GCS equivalent later sits beside them rather than replacing them. **S3** uses the same storage contract as the local backend, identical layout and identical code paths, so switching is one config line and a laptop run can publish to the shared bucket. Writes are immutable: a conditional put means two sessions racing for one key cannot silently overwrite each other. reportr reads from the same place and publishes the site beside the results, so a reader who clicks through to a data point gets the real report. **EC2** provisioning creates a security group, an ephemeral key pair, N Solr instances and one driver per session, runs the matrix over SSH, and destroys all of it. Solr runs in Docker with a standalone ZooKeeper co-located on the first node and the index on its own volume, since the gp3 root baseline is slow enough to measure the disk instead of Solr. Host tuning is applied and then read back rather than assumed. Everything is tagged, and an append-only ledger is written to storage before anything exists, so `launchr teardown` can reconcile tags against the ledger and reclaim what a crashed session left behind — each covers the other's blind spot. **Two dependency choices worth calling out.** Provisioning uses boto3 rather than Pulumi: the Pulumi Automation API shells out to the `pulumi` CLI, which would have made an external executable a hard runtime dependency. Storage uses boto3 directly rather than `cloudpathlib`, keeping the dependency surface to the one library EC2 already needs — the trade-off being that an Azure or GCS backend becomes its own implementation of the same Protocol rather than falling out of a shared path abstraction. Also adds a Continuous Benchmarking chapter to the docs, command references for both tools, and working AWS configurations under `examples/`. Verified against a real AWS account: a two-node SolrCloud forms, results are written to S3, the site publishes and renders, and provision and teardown recover correctly from a crashed session. One limitation worth stating plainly: no full-corpus run has completed yet. Three attempts on the full nyc_taxis dataset all failed at `optimize`, for two reasons that turned out to be pre-existing and are now fixed separately in #115 and #117. The verification above is therefore from `--test-mode` runs plus the infrastructure paths exercised at full size; a full-dataset run on top of those fixes is the next step. Stacked on #108 and #109 — review those first; this branch contains both. Fixes #129 🤖 Assisted by Claude Code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
