Nick Shanin created SOLR-18506:
----------------------------------
Summary: TestThinCache.testSimple is flaky: eviction assertion
races Caffeine removal notification delivery
Key: SOLR-18506
URL: https://issues.apache.org/jira/browse/SOLR-18506
Project: Solr
Issue Type: Bug
Reporter: Nick Shanin
AI-generated, human-approved text below.
org.apache.solr.search.TestThinCache.testSimple fails intermittently with
"AssertionError: expected:<0> but was:<1>" at TestThinCache.java:170, together
with a classMethod teardown error reporting 3 unreleased SolrMetricsContext
objects. The test is the only coverage of ThinCache, the per-searcher scope
over a shared node-level cache introduced in SOLR-16654. The failure has been
recorded 14 times in the last 30 days across unrelated pull requests (list
below), including 10 times on branch_9x. It is a test defect, not a product
defect: no production code is involved in the race.
Failure signature
- TestThinCache.testSimple: expected:<0> but was:<1> at TestThinCache.java:170
(the second cache's evictions metric). The value is exactly 1 on every recorded
occurrence.
- classMethod teardown: 3 unreleased SolrMetricsContext objects (knock-on, see
root cause).
- Recorded occurrences (GitHub "Solr Tests via Crave" runs, 2026-09-09 to
2026-10-04): PR #4894 (seed C4077EAB80AE9E50), PR #4939 (60B12BB09380AC51), PR
#4943 (E58B5E1FFDEBA340), PR #5013 (30A1B1A830109349), and PR #4976 on
branch_9x ten times (E4DD10799981D14B, 2E53727468DE4DF1, 391A297131D7D9C1,
D8FD97BF9A8A48B0, FCA68807697335BA, 861F48CD4DFF5F51, D5B22838079384ED,
6E0A7AD71C551639, 7A1C354A4C7E10DC, FA416BA281EBD0DF).
Reproduction
Locally, the failure does not reproduce: all 14 recorded seeds were run against
current main with forced re-execution (cleanTest), plus default-seed runs, and
every run passed. The #5013 seed also passed earlier on pre-merge main, current
main, and the PR head. The race needs the timing of a loaded CI runner; the
mechanism below is established from the code and from the invariant failure
value.
Root cause
testSimple puts 101 entries through a first ThinCache into a shared backing
cache of capacity 100, creating exactly one eviction (key 1; the test asserts
its physical absence). It then warms a second ThinCache from the first.
ThinCache.warm copies the first cache's counters into the second cache's
priors, including evictions. The second cache's eviction metric is cumulative
and prior-inclusive, like its hits, inserts, and lookups in the same assertion
block (4 = 2 + 2, 102 = 101 + 1, 7 = 4 + 3), so the inherited value is 1. The
assertion expects 0, which holds only if Caffeine has not yet delivered the
first eviction's removal notification when warm copies the counter. Physical
eviction and notification delivery are separate steps in Caffeine (the backing
cache is an async Caffeine cache whose maintenance runs lazily), and the test
has no synchronization point between them, so the sample races the delivery.
The teardown leak is a knock-on: the failed assertion skips the test's closing
of its SolrMetricsContext, and the framework reports the context and its two
per-cache children as unreleased.
Proposed fix
Test-only, in testSimple: enlarge the shared backing cache (public
CaffeineCache.setMaxSize, which also forces Caffeine maintenance via cleanUp)
after the first cache's assertions and before warming, so the pending eviction
is settled and counted before the priors copy and no further eviction can occur
in the second phase; assert the first cache's eviction counter is exactly 1 at
that point; and correct the second cache's expected evictions from 0 to the
inherited 1, with a comment explaining the cumulative accounting. A pull
request with this fix and a seed battery proof will follow.
### AI assistance
AI agents assisted with research, implementation, review, and drafting. Nick
Shanin directed the work and takes responsibility for this ticket description.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
