lxy-9602 opened a new pull request, #484: URL: https://github.com/apache/paimon-cpp/pull/484
<!-- PR titles must follow Conventional Commits: <type>(<optional-scope>): <description> --> ### Purpose <!-- Linking this pull request to the issue --> Linked issue: #158 Improve real-time reads, spill behavior, and index support. - Resolve historical file schemas for offset lookup, supporting compatible key renames and field reordering after recreating the context. - Trim pending DV offsets per file using schema-aware statistics, with conservative fallback when statistics are unavailable. - Compose input selection bitmaps with offset deletions, use ordered offset traversal, and allow sparse input bitmaps. - Enable deduplicate spill through two-phase sealing: freeze data, key indexes, and DV checkpoints atomically, then spill outside the write mutex. - Freeze without temporary IPC spill during `PrepareCommit`; retain spill behavior for explicit Seal. - Include key indexes and distinct DV bitmaps in memory accounting, and release retired resources outside critical locks. - Prepare committed snapshot refresh outside progress locks and reuse unchanged sidecar readers. - Replace the bitmap-only restriction with exact BIGINT `IN`/`NOT_IN` capability validation, supporting bitmap, BSI, and range-bitmap. - Reject non-main-branch real-time operations. <!-- What is the purpose of the change --> ### Tests Added or updated coverage for: - Historical schemas, offset-stat projection, and conservative fallback. - Bitmap composition, sparse selections, and ordered offset filtering. - Concurrent writes during spill, checkpoint stability, memory accounting, and resource cleanup. - Snapshot refresh concurrency, failure/retry, and reader reuse. - Embedded/external exact indexes and prepare-without-spill across append, deduplicate, PK MOR, and PK DV. <!-- List UT and IT cases to verify this change --> ### API and Format - Add optional `RealtimeStoreTwoPhaseSeal`; custom stores retain the legacy path, while deduplicate spill requires this capability. - Add `FileIndexer::SupportsExactPredicate`, defaulting to false. Binary plugins implementing `FileIndexer` require rebuilding. - Keep capability validation at the configuration/schema level; existing-file consistency remains the application’s responsibility. <!-- Does this change affect API in include dir or storage format or protocol --> ### Documentation Add a real-time guide covering supported modes, configuration, pluggable stores, read/commit lifecycle, indexes, spill, memory accounting, and limitations. <!-- Does this change introduce a new feature --> ### Generative AI tooling Generated-by: OpenAI Codex <!-- If generative AI tooling has been used in the process of authoring this patch, please include the phrase: 'Generated-by: ' followed by the name of the tool and its version. If no, write 'No'. Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details. --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
