nick-boss-tech commented on PR #5323: URL: https://github.com/apache/solr/pull/5323#issuecomment-6865588819
🤖 *AI text below* 🤖 *(reviewed by Nick Shanin)* **TL;DR:** I found a few gaps to address before CATS replaces the automatic BATS coverage: authenticated uploads need result checks, TLS tests need to distinguish the intended rejection from unrelated failures, the resource-warning regression case needs a port, and rolling-upgrade CI needs to verify that the test actually ran. I explored these on an earlier head in [this follow-up diff](https://github.com/nick-boss-tech/solr/compare/73aed042f6e2c46dfddad0f086234136d5daa299...e92e417843e24dac8a54bd1c3c61d43e7b420cf4). Since then, the rolling-upgrade test has moved to the Docker module, so that part of the diff is historical. <details> <summary><strong>Authenticated posting and log indexing</strong></summary> `BasicAuthTest` checks for `1 files indexed.` and `Committed`, but does not query for the posted document or indexed logs. Those messages can appear even when the intended documents are absent. In an earlier control run, log uploads received HTTP 503 while the commit succeeded and the old assertion passed. The [follow-up test](https://github.com/nick-boss-tech/solr/blob/e92e417843e24dac8a54bd1c3c61d43e7b420cf4/solr/packaging/src/test/org/apache/solr/shell/BasicAuthTest.java#L110) queries for both results and rejects `Batch sending failed:`. On Linux with JDK 21 and 25, BasicAuth passed 4/4 at the follow-up head. With tests unchanged, suppressing the post commit made the document-count assertion fail; changing the indexed log field while allowing the commit to succeed made the log query fail. </details> <details> <summary><strong>TLS rejection causes</strong></summary> Some SSL cases accept a generic connection failure, and the peer-name assertion accepts `Server refused connection`. An unavailable node can therefore satisfy an assertion intended to demonstrate a TLS or hostname rejection. The mTLS cases also need care: a client can report a closed channel while the more specific failure appears on the server. The [follow-up](https://github.com/nick-boss-tech/solr/blob/e92e417843e24dac8a54bd1c3c61d43e7b420cf4/solr/packaging/src/test/org/apache/solr/shell/SslTest.java#L103) adds diagnostic fixtures that reject refusal, stale log entries, and unrelated certificate failures. The fixtures passed on JDK 21 and 25; mutations that weakened their refusal and stale-log checks failed at the intended assertions. The live needed-mTLS case is still marked `@AwaitsFix`. When enabled, it stopped because the client reported a closed channel where the test expected a certificate alert. The matching/mismatching client sequence therefore did **not** complete, so the fixture results do not establish that live hostname verification is correct. </details> <details> <summary><strong>Example resource-limit warnings</strong></summary> The BATS example suite checks that ordinary starts retain resource-limit warnings while example starts suppress them. I could not find the equivalent case in the current `ExampleTest`. Since the BATS workflow becomes manual-only, this behavior would leave the automatic checks. The [ported case](https://github.com/nick-boss-tech/solr/blob/e92e417843e24dac8a54bd1c3c61d43e7b420cf4/solr/packaging/src/test/org/apache/solr/shell/ExampleTest.java#L79) passed as part of ExampleTest, 3/3 on JDK 21 and 25. Removing the example-mode warning suppression made the new assertion fail. </details> <details> <summary><strong>Rolling-upgrade execution</strong></summary> I saw that the rolling-upgrade test has moved to the Docker module, with CI wiring tracked in SOLR-18658. It would be useful for that wiring to build the candidate image and fail when the upgrade test is missing or skipped, so a successful job establishes that the upgrade actually ran. The Linux host used for the follow-up did not have Docker, so I have **not** verified a rolling upgrade with the candidate image. </details> <details> <summary><strong>Filtered process discovery</strong></summary> A marker-filtered `SolrProcessManager` scan can treat a live node outside the filter as stale and delete its PID or port file if the nodes share a PID directory. CATS normally gives each class a separate directory, but an inherited `SOLR_PID_DIR` can override that isolation. The [follow-up test](https://github.com/nick-boss-tech/solr/blob/e92e417843e24dac8a54bd1c3c61d43e7b420cf4/solr/core/src/test/org/apache/solr/cli/SolrProcessManagerTest.java#L158) checks matching and nonmatching live nodes in both file formats. It passed on the follow-up head; with the contributor’s earlier production code restored, it failed at the foreign PID-file assertion. A separate empty-result mutation failed at the expected result-count assertion. </details> The [Linux evidence](https://github.com/nick-boss-tech/solr/blob/dbca42266e8d3e99bd8eb3b938684c11a1513c7b/records/evidence/pr5323-tls-isolation-round4-linux-evidence.txt) has the test commands, mutations, and execution conditions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
