mikebridge opened a new pull request, #41824: URL: https://github.com/apache/superset/pull/41824
### SUMMARY Contribution **into #40221's branch** (`sl-cache`), addressing @rusackas's twice-flagged review finding: `store_result` performs a non-atomic read-modify-write on the index bucket, so two concurrent writers for one view can clobber each other's entries. The eviction prune in `try_serve_from_cache` has the same shape and can *actively drop* an entry added between its read and write. **Why not WATCH/MULTI or Lua** (the original suggestion): the index value is a pickled Python list managed through the Flask-Caching abstraction, and `data_cache` is not guaranteed to be Redis-backed. Lua can't merge pickled payloads, and `WATCH` requires reaching into Redis-only client internals. **The fix**: a short-lived lock key acquired via `cache.add` — the portable set-if-absent primitive (`SET NX` on Redis, `add` on memcached) — serialises index writers per bucket, with a hard invariant: *when the backend supports `add()`, the index is only ever written while holding the lock*. On acquisition timeout (max ~500 ms) the update is **skipped** — losing at most that write's future cache benefit — rather than performed unlocked, which could clobber a concurrent writer. Backends without `add()` degrade to the previous behavior. Both `store_result` and the prune path go through the same helper (`_update_index`), and the prune now re-reads under the lock instead of writing back a stale list. ### TESTING INSTRUCTIONS `test_concurrent_store_results_do_not_clobber_index` choreographs the exact historical interleaving with events — **no timing dependence**: thread A is parked inside its index read while thread B runs a full `store_result` for the same view, then A resumes. Validated bidirectionally: - **Without the fix**: fails every run (3/3) with `B's entry was clobbered by A's stale index write`. - **With the fix**: passes every run — B blocks on the lock, releases A, then appends; both entries survive. ```bash pytest tests/unit_tests/semantic_layers/ -q # 374 passed ``` ### ADDITIONAL INFORMATION - [ ] Has associated issue: - [ ] Required feature flags: - [ ] Changes UI - [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351)) - [ ] Introduces new feature or API - [ ] Removes existing feature or API 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
