nagisa-kunhah commented on PR #3567:
URL: https://github.com/apache/kvrocks/pull/3567#issuecomment-5152182677
Hi @jihuayu, while implementing `CF.DEL`, I found that the automatic
compaction path may introduce serious worker-blocking and OOM risks. Currently,
it scans all logical buckets of a sub‑filter and loads every page into
`CuckooPageCache`, including zero‑filled pages for missing keys. Thus, a large
sparse sub‑filter can make a single `CF.DEL` perform work and allocate memory
proportional to its logical capacity.
We seek feedback on whether compaction should be part of this PR. Two
options:
1. **Land `CF.DEL` without automatic compaction for now.**
Only remove fingerprint and update metadata. Semantics are preserved, but
deleted slots and extra sub‑filters remain, causing space and lookup
amplification.
2. **Move compaction to bounded, incremental background maintenance.**
Use Kvrocks’ `TaskRunner` (used by `AsyncCompactDB`, `AsyncScanDBSize`,
etc.) with cursor‑based scanning. Process a bounded number of actually stored
pages per slice, commit a bounded write batch, release cache and locks, and
requeue if more work remains. Requires actual‑page iteration, bounded
cache/batch, deduplication, synchronization, and crash‑safe progress tracking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
