lizhimins opened a new pull request, #10335: URL: https://github.com/apache/rocketmq/pull/10335
## Summary - Add native C++ `CqCompactionFilter` JNI shim for ConsumeQueue compaction, supporting Linux, macOS (x86_64/aarch64), and Windows platforms - Replace the Java-based `ConsumeQueueCompactionFilterFactory` with a native compaction filter that runs entirely in RocksDB's C++ compaction threads, avoiding JNI callback overhead per key - Enhance `manualCompaction` logging with `estimateNumKeys` before/after and reduction ratio - Remove unused `minPhyOffset` parameter from `AbstractRocksDBStorage.manualCompaction` ## Motivation The existing Java `CompactionFilterFactory` creates a `RemoveConsumeQueueCompactionFilter` per compaction job, which incurs JNI cross-boundary calls for every key during compaction. For ConsumeQueue databases with millions of entries, this overhead is significant. A native C++ compaction filter: 1. Runs entirely in RocksDB's native compaction threads — zero JNI overhead per key 2. Uses atomic `minPhyOffset` threshold updated from Java via a single JNI call 3. Supports all major platforms (Linux, macOS, Windows) ## Changes 1. **`CqCompactionFilterJni.java`** — JNI loader with platform detection, RocksDB native library co-location for `DT_NEEDED`/`LC_LOAD_DYLIB` resolution 2. **`NativeCqCompactionFilter.java`** — Thin wrapper with `disOwnNativeHandle()` to prevent use-after-free when options close before DB 3. **`cq_compaction_filter.cpp`** — Native C++ filter implementation with `std::atomic<int64_t>` threshold and JNI exports 4. **Pre-built binaries** — `libcq_compaction_filter.so` (Linux), `.dylib` (macOS) 5. **`AbstractRocksDBStorage`** — Enhanced compaction logging, removed unused `minPhyOffset` param 6. **`ConsumeQueueRocksDBStorage`** — Integrates native filter on startup with graceful fallback 7. **`docs/en/Native_RocksDB.md`** — Build instructions for all platforms ## Test Plan - [x] `CqCompactionFilterJniTest` — verifies native library loading, filter creation, minPhyOffset update, and actual compaction filtering - [x] Existing `ConsumeQueueRocksDBStorage` integration tests pass - [ ] Manual verification on macOS and Windows Closes #10334 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
