janhoy opened a new issue, #107: URL: https://github.com/apache/solr-orbit/issues/107
Nothing in CI runs `it/`. The five workflows are `unit-test.yml`, `docker-build.yml`, `dockerfile-lint.yml`, `docs-lint.yml` and `check-actions-usage.yml`, and `unit-test.yml` runs `.ci/build.sh build_and_unit_test` — unit tests only. Unguarded, the suite rotted until it could not run at all. Everything fixed in #105 would have been caught the week it broke: a metrics store pinned to a nonexistent "Solr 1.3.9", a `DISTRIBUTIONS` entry for an unreleased Solr 10.1.0, a workloads revision deleted from the repository, `--opensearch-plugins`, `--runtime-jdk=bundled`, and a call to a `client.wait_for_rest_layer` that does not exist. It also hid the fact that the suite invoked a bare `solr-orbit` from `PATH`, so it was testing whatever copy happened to be installed rather than the working tree. **Proposal: a `workflow_dispatch` job on Linux.** Manually triggered rather than on every PR, because the suite downloads Solr distributions, builds from source and runs real benchmarks — minutes to tens of minutes, and too slow and too flaky-prone to gate every push. Manual is enough to make rot visible, and a maintainer can run it before a release or when touching the builder. Sketch: - `workflow_dispatch` with an optional input to select a subset (e.g. `it/info_test.py it/list_test.py`), defaulting to the whole suite. - `ubuntu-latest`. Linux also sidesteps the macOS Thespian hangs described in #106, so the cluster-provisioning tests can actually run. - JDK 21 (Solr 10 requires it), Docker and `docker-compose` available on the runner. - `make it`, or `uv run pytest -s it` to skip the tox layer. - Upload the junit XML and `~/.solr-orbit/logs/` as artifacts, since these failures are usually only diagnosable from the logs. - Pin every third-party action to a full commit SHA, per `check-actions-usage.yml`. Worth considering as a follow-up: a nightly schedule once it is reliably green, and a separate cheap job that builds the wheel, installs it into a clean venv and runs `solr-orbit list cluster-configs` — the packaging breakage fixed in #100 was invisible to every existing check. 🤖 Assisted by Claude Code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
