Nick Shanin created SOLR-18609:
----------------------------------
Summary: Packaging BATS tests accept failures they should catch:
harden the assertions
Key: SOLR-18609
URL: https://issues.apache.org/jira/browse/SOLR-18609
Project: Solr
Issue Type: Test
Reporter: Nick Shanin
A review of the packaging BATS suites ({{solr/packaging/test}}) found
assertions that pass when the operation they test has failed. The suites stay
green while the behavior they guard is broken.
h3. What the review found
* Post tests that never verified the posted documents arrived, including an
authenticated post aimed at the wrong collection name.
* A log batch whose sends failed could still pass after an empty commit; the
check did not require any indexed records.
* Failed ZooKeeper reads were accepted as proof that a child node was deleted
(seven negative existence checks).
* Degraded collection health was accepted: shard statuses were read without
requiring command success and the collection-level healthy status.
* A failed stop was accepted as a graceful stop, and a stale export output
file could pass as a fresh result.
* TLS failure checks matched so generically that a plain connection refusal
satisfied them; a failed curl was accepted as a compression check.
h3. Control evidence
With an HTTP 503 injected into a log batch add, the current authenticated post
test passes 1/1 while the follow-up probes show zero documents indexed; the
hardened test rejects the same injection. The hardened assertions require
command success, the specific failure diagnostic where one exists, and a
positive check that the expected data or state is present.
h3. Prepared branch
Branch:
[solr-bats-assertion-hardening-submit|https://github.com/nick-boss-tech/solr/tree/solr-bats-assertion-hardening-submit]
at {{4d5d876d946a4f5ac174a7f68e16f4dd9b5f9f68}}, 14 files under
{{solr/packaging/test}} (13 existing files and the new
{{test_bats_assertions.bats}} with nine helper regression cases). Test files
only; no production, launcher, or build changes.
Diff: [base to
branch|https://github.com/nick-boss-tech/solr/compare/0d6b2e32f598d772aed094038022a6a588fdfc5a...4d5d876d946a4f5ac174a7f68e16f4dd9b5f9f68]
This generalizes the one-line fix merged under SOLR-18565 (PR #5193) and the
assertion direction of the author's own BATS work; it is filed separately
because its scope is the whole suite, not the single test_ssl case.
h3. Verification and limits
Proven on Linux: the Picocli-selected cases pass 18/18 on JDK 21 and on JDK 25;
the batch-failure controls above pass (old test accepts the injection, hardened
test rejects it); tidy is clean and {{:solr:packaging:check -x test}} passes.
Not yet proven, stated plainly: the legacy TLS legs show connection-refused
failures in the sandbox they ran in, the client-hostname policy control needs a
host with more tmpfs headroom than any host available here, and
{{test_rolling_upgrade.bats}} needs Docker, which is not available here. An
upstream CI run is the natural place to settle those three.
🤖 _AI text above_ 🤖 _(reviewed by Nick Shanin)_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
