nagisa-kunhah commented on PR #3567: URL: https://github.com/apache/kvrocks/pull/3567#issuecomment-5306198245
# CF.DEL Background Compaction `CF.DEL` must not compact a complete logical sub-filter on the request path. A large sparse filter can have very few RocksDB pages but a huge logical page range, so a full logical scan can block a worker and materialize excessive page-cache memory. ## Compaction For the current tail `Fn`, compact oldest targets first: ```text Fn -> F0 remaining Fn -> F1 ... remaining Fn -> F(n - 1) verify Fn ``` For each non-zero fingerprint in `Fn`, try direct insertion into the two candidate buckets of the current target. If insertion succeeds, write the target fingerprint and clear the source fingerprint in one `WriteBatch`. Do not use Cuckoo kick-out relocation during compaction. Scan `Fn` through its persisted page-key prefix, so sparse filters do not trigger reads for absent logical pages. Delete a source page key when its page becomes all zero. One `Fn -> Fi` pass holds the existing logical-key lock. It may flush bounded page batches while keeping that lock. After a pass completes, release the lock, then start the next target pass. If `Fn` is empty after all targets were tried, decrement `n_filters`. If that exposes another tail, continue with it in the same task. ## Task execution and deduplication A successful `CF.DEL` schedules compact work when the chain has more than one sub-filter and delete debt exceeds the compact threshold. The task carries: | Field | Description | |---|---| | `namespace` and `user_key` | Reconstruct the Cuckoo Filter in the background callback. | | `ns_key` | Lock key and task-key input. | | `metadata_version` | Reject a key that was deleted and recreated before the task starts. | `TaskRunner` should gain a generic keyed API: ```cpp StatusOr<bool> TryPublishUnique(std::string task_key, Task task); ``` The compact task key is: ```text cuckoo-compact:<ns_key>:<metadata_version> ``` `TaskRunner` keeps active task keys in memory. The same key is queued only once while it is queued or running. It removes the key when the callback returns, rolls it back if queueing fails, and clears all keys when `Join()` clears the queue. The callback locks `ns_key`, rereads metadata, and exits if `metadata_version` no longer matches. A restart drops queued work and active task keys. It does not resume compact work. A later qualifying `CF.DEL` can schedule a new best-effort task. ## Trade-offs - A full `Fn -> Fi` pass can delay foreground writes for that key. - One tail is scanned once per older target, and target bucket reads can be random. - `CF.ADD` between target passes can make the current cycle less effective, but cannot make tail removal unsafe because the final empty check is locked. - `TaskRunner` is shared and single-threaded by default, so long compact tasks can delay other background work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
