rishabhdaim opened a new pull request, #3071:
URL: https://github.com/apache/jackrabbit-oak/pull/3071
## Summary
`CacheBuilder` configured Caffeine with `executor(Runnable::run)`, forcing
all cache maintenance
(eviction, weight accounting, removal callbacks, read-buffer drains) to run
inline on whichever
thread triggered it — while holding Caffeine's global eviction lock. This
produced three
symptoms traced to the same root cause:
- **OAK-12290** — with `FT_OAK-12214` on, every L1 segment hit calls
`cache.getIfPresent()`
(`SegmentCache.recordHit`). That periodically wins the CAS in `afterRead`
→ `scheduleDrainBuffers`
→ `performCleanUp`, so HTTP request threads serving pages ran full cache
maintenance mid-request
and slowed all concurrent segment-cache operations on the pod.
- **Publish JVM wedge** — `FileStore.writeSegment` →
`SegmentCache.putSegment` → `afterWrite` →
`lock()`. The async indexer thread holding the lock died; `ReentrantLock`
has no dead-owner
detection, so 20+ threads (TarMK flush, Sling jobs, HTTP workers) queued
behind it permanently
and the pod needed a restart.
- **SAML login slowness** — concurrent cache misses each ran eviction
maintenance inline after a
TAR read, backing up on the eviction lock and emitting Caffeine's
`excessive wait times for acquiring the eviction lock` warning.
The fix lets Caffeine dispatch maintenance on its own executor — the
documented default — so no
request, indexer or writer thread holds the eviction lock inline.
## Changes
- `CacheBuilder.configureCaffeineBuilder()` — no longer installs the
caller-runs executor.
Gated by a bug-fix feature toggle (`FT_OAK-12290`, default on) with a
`-Doak.cache.asyncMaintenance=false` escape hatch, since the toggle is
read at cache-build time.
- `SegmentNodeStoreRegistrar` — registers the new toggle on the Whiteboard
next to `FT_OAK-12214`.
- `cache/api/package-info.java` — package version 1.0.0 → 1.1.0 for the new
exported constants
(OSGi baseline).
- `CacheBuilderMaintenanceTest` (new) — reproduces the issue: asserts
maintenance does not run on
the calling thread, that a slow maintenance callback does not block the
writer, and that
disabling the toggle restores inline behaviour.
- `CacheBuilderTest` / `SegmentCacheTest` — eviction callbacks are now
asynchronous, so
assertions on post-eviction state await the callback instead of assuming
it ran inline.
## Test Plan
- [x] New `CacheBuilderMaintenanceTest` fails on trunk (both assertions) and
passes with the fix
- [x] `oak-core-spi` — 220 tests pass
- [x] `oak-segment-tar` — 817 tests pass
- [x] `oak-core` — 4772 tests pass
- [x] `oak-store-document` — 2210 tests pass
- [x] OSGi baseline check passes
## Links
- JIRA: https://issues.apache.org/jira/browse/OAK-12290
- Related: https://issues.apache.org/jira/browse/OAK-12214,
https://issues.apache.org/jira/browse/OAK-12197
Co-Authored-By: Claude Opus 5 <[email protected]>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
