bobhan1 opened a new pull request, #438: URL: https://github.com/apache/doris-thirdparty/pull/438
### Background Index compaction translates source document IDs into destination index/document pairs. Its dense matrix parameter is copied into IndexWriter, preventing callers from supplying bounded compressed mappings such as the storage introduced by apache/doris#70350. ### What problem does this PR solve? Issue Number: None Related PR: apache/doris#70350 (compressed storage), apache/doris#69313 and #420 (existing token-statistics prerequisite) PR Type: Performance Problem Summary: Add a caller-owned `DocIdMapping` interface and an `indexCompaction` overload that borrows it during synchronous compaction. Postings, both norms paths and null bitmaps use fallible point lookups, without depending on Doris headers or materializing a dense matrix. The existing dense overload and `setTransVec` remain available through an owned dense adapter. ### Details - **Change map:** 253 production lines changed and 160 test lines across four files; no generated files or new dependencies. - **Why it was slow:** The old API requires a full matrix and copies its row pairs into the writer. The matrix alone is 8 bytes per physical source row, before the extra copy and other compaction allocations. - **Optimization:** Borrow a provider for the new entry. Terms may revisit lower source doc IDs; directory positions and the existing deleted-row sentinel are preserved. The borrowed pointer resets on success and exception. Diagnostic source locations come from the queued posting rather than a reverse scan of the matrix. - **Behavior change:** Successful results and the legacy dense entry remain compatible. Lookup failures abort compaction and release the resources acquired by the new provider path. - **Failure handling:** A lookup error propagates as `CLuceneError`, never as a deleted document. ASAN failure injection exposed leaks after output creation and inside posting merge. RAII now owns queued/active source iterators, destination writers and segment metadata; norms/null outputs join the existing finally cleanup. All three lookup-failure phases are covered. - **Results:** Independent ASAN/LSAN compaction tests pass 9/9. The coordinated local Doris stack passes 67/67 targeted ASAN tests and a complete PCH-disabled production Release BE build. These are correctness/build results; no optimized throughput or billion-row compaction measurements were repeated for this change. - **Applicability and cost:** One virtual call per lookup; a document can be revisited by different terms or fields. Representation and cache policy belong to the caller. The legacy dense entry retains its ownership/copying behavior; clients adopt the new entry to avoid the copy. IndexWriter layout changes require clients to rebuild. ### Testing Independent CLucene build with Clang 20.1.8 and `CMAKE_BUILD_TYPE=ASAN`, using the repository's CMake target: ```sh cmake --build <build-dir> --target cl_test --parallel 100 cd <build-dir>/bin ./cl_test -t IndexCompaction ``` All nine tests pass with ASAN/LSAN and no new suppressions: seven existing cases, borrowed/dense equivalence for postings/positions, norm bytes, token totals and null row IDs, source rewinds, diagnostic output, and lookup failures in norms, postings and null-only remapping. The all-phase failure test leaks with the earlier raw ownership and passes after the cleanup fix. The local Doris integration uses master `a02ea2b6572e51fc19988f268c1a6fe1274d7286`, the production eligibility hunk from the still-open apache/doris#69313, and the new Doris adapter. That prerequisite is required by the norms/frequency guard already merged in #420; it is not part of this interface PR. No Doris adoption PR has been opened yet. ```sh bash run-be-ut.sh --run --filter='CluceneRowIdMappingTest.*:*TestRowIdConversion*:CompressedRowIdMapTest.*:IndexCompactionTest.*:IndexCompactionWriteIndexTest.*:SniiRowIdConversionTest.*-IndexCompactionTest.tes_wikipedia*' -j100 ENABLE_PCH=OFF ENABLE_UNITY_BUILD=ON bash build.sh --be --coverage -j100 ``` Doris tests cover bounded cross-block/backward lookups, malformed input and lookup errors, dense/compressed in-memory index equivalence, and real Duplicate Key/MoW compaction UTs that verify COMPRESSED mode and compare against raw rebuilds. The production build passes compilation, linking, installation and packaging with `MAKE_TEST=OFF`; the packaged binary meets the GLIBC 2.17 baseline. clang-format 16, build hygiene and whitespace checks pass. ASAN lookup timing output is diagnostic only. The final targeted run includes both compressed-mode pipeline tests and representative legacy cases; the ten long legacy wiki variants are excluded from that final rerun after correcting the test helper to call production mapping selection/finalization. ### Release note Index compaction can borrow a document-ID mapping provider without requiring a dense matrix. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No; interface ownership and lookup semantics are documented in source. - Pre-submission status: Ready ### AI Disclosure - Tool: OpenAI Codex (GPT-6) - Generated: Implementation, tests and description. - Human review: Pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
