nick-boss-tech commented on PR #5323:
URL: https://github.com/apache/solr/pull/5323#issuecomment-6865588819

   🤖 *AI text below* 🤖 *(reviewed by Nick Shanin)*
   
   **TL;DR:** I found a few gaps to address before CATS replaces the automatic 
BATS coverage: authenticated uploads need result checks, TLS tests need to 
distinguish the intended rejection from unrelated failures, the 
resource-warning regression case needs a port, and rolling-upgrade CI needs to 
verify that the test actually ran.
   
   I explored these on an earlier head in [this follow-up 
diff](https://github.com/nick-boss-tech/solr/compare/73aed042f6e2c46dfddad0f086234136d5daa299...e92e417843e24dac8a54bd1c3c61d43e7b420cf4).
 Since then, the rolling-upgrade test has moved to the Docker module, so that 
part of the diff is historical.
   
   <details>
   <summary><strong>Authenticated posting and log indexing</strong></summary>
   
   `BasicAuthTest` checks for `1 files indexed.` and `Committed`, but does not 
query for the posted document or indexed logs. Those messages can appear even 
when the intended documents are absent. In an earlier control run, log uploads 
received HTTP 503 while the commit succeeded and the old assertion passed.
   
   The [follow-up 
test](https://github.com/nick-boss-tech/solr/blob/e92e417843e24dac8a54bd1c3c61d43e7b420cf4/solr/packaging/src/test/org/apache/solr/shell/BasicAuthTest.java#L110)
 queries for both results and rejects `Batch sending failed:`.
   
   On Linux with JDK 21 and 25, BasicAuth passed 4/4 at the follow-up head. 
With tests unchanged, suppressing the post commit made the document-count 
assertion fail; changing the indexed log field while allowing the commit to 
succeed made the log query fail.
   
   </details>
   
   <details>
   <summary><strong>TLS rejection causes</strong></summary>
   
   Some SSL cases accept a generic connection failure, and the peer-name 
assertion accepts `Server refused connection`. An unavailable node can 
therefore satisfy an assertion intended to demonstrate a TLS or hostname 
rejection. The mTLS cases also need care: a client can report a closed channel 
while the more specific failure appears on the server.
   
   The 
[follow-up](https://github.com/nick-boss-tech/solr/blob/e92e417843e24dac8a54bd1c3c61d43e7b420cf4/solr/packaging/src/test/org/apache/solr/shell/SslTest.java#L103)
 adds diagnostic fixtures that reject refusal, stale log entries, and unrelated 
certificate failures. The fixtures passed on JDK 21 and 25; mutations that 
weakened their refusal and stale-log checks failed at the intended assertions.
   
   The live needed-mTLS case is still marked `@AwaitsFix`. When enabled, it 
stopped because the client reported a closed channel where the test expected a 
certificate alert. The matching/mismatching client sequence therefore did 
**not** complete, so the fixture results do not establish that live hostname 
verification is correct.
   
   </details>
   
   <details>
   <summary><strong>Example resource-limit warnings</strong></summary>
   
   The BATS example suite checks that ordinary starts retain resource-limit 
warnings while example starts suppress them. I could not find the equivalent 
case in the current `ExampleTest`. Since the BATS workflow becomes manual-only, 
this behavior would leave the automatic checks.
   
   The [ported 
case](https://github.com/nick-boss-tech/solr/blob/e92e417843e24dac8a54bd1c3c61d43e7b420cf4/solr/packaging/src/test/org/apache/solr/shell/ExampleTest.java#L79)
 passed as part of ExampleTest, 3/3 on JDK 21 and 25. Removing the example-mode 
warning suppression made the new assertion fail.
   
   </details>
   
   <details>
   <summary><strong>Rolling-upgrade execution</strong></summary>
   
   I saw that the rolling-upgrade test has moved to the Docker module, with CI 
wiring tracked in SOLR-18658. It would be useful for that wiring to build the 
candidate image and fail when the upgrade test is missing or skipped, so a 
successful job establishes that the upgrade actually ran.
   
   The Linux host used for the follow-up did not have Docker, so I have **not** 
verified a rolling upgrade with the candidate image.
   
   </details>
   
   <details>
   <summary><strong>Filtered process discovery</strong></summary>
   
   A marker-filtered `SolrProcessManager` scan can treat a live node outside 
the filter as stale and delete its PID or port file if the nodes share a PID 
directory. CATS normally gives each class a separate directory, but an 
inherited `SOLR_PID_DIR` can override that isolation.
   
   The [follow-up 
test](https://github.com/nick-boss-tech/solr/blob/e92e417843e24dac8a54bd1c3c61d43e7b420cf4/solr/core/src/test/org/apache/solr/cli/SolrProcessManagerTest.java#L158)
 checks matching and nonmatching live nodes in both file formats. It passed on 
the follow-up head; with the contributor’s earlier production code restored, it 
failed at the foreign PID-file assertion. A separate empty-result mutation 
failed at the expected result-count assertion.
   
   </details>
   
   The [Linux 
evidence](https://github.com/nick-boss-tech/solr/blob/dbca42266e8d3e99bd8eb3b938684c11a1513c7b/records/evidence/pr5323-tls-isolation-round4-linux-evidence.txt)
 has the test commands, mutations, and execution conditions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to