SteNicholas opened a new pull request, #191: URL: https://github.com/apache/paimon-cpp/pull/191
### Purpose Linked issue: close #169 Port the read side of Apache Paimon [`d2be7eace`](https://github.com/apache/paimon/commit/d2be7eacedf7c001ed1022856a62ade089e954c4) (apache/paimon#8380), so a data-evolution table can enable `deletion-vectors.enabled`. A row range group's deletion vector is maintained against the group's **anchor file** — the oldest normal file, compared by `(max_sequence_number, file_name)`, skipping blob and vector-store files — so its positions are anchor-relative. Every reader of the group must therefore drop the same rows to keep the column merge positionally aligned: - file readers apply the vector shifted by the file's offset inside the anchor row id range; - the blob fallback path drops the deleted row ids from its placeholder gap segments, which have no file reader to wrap. Main changes: - Add `DataEvolutionUtils::RetrieveAnchorFile`, matching Java's anchor rule. The producer side (`AppendDeleteFileMaintainer`) documents the same contract, so the file a Java engine attaches the deletion vector to is the one this read path looks it up by. - Drop the `SchemaValidation` rule forbidding `data-evolution.enabled` together with `deletion-vectors.enabled`, matching Java. - Subtract deletion vector cardinality from `DataSplit::MergedRowCount`, and report the count unavailable when a deletion file carries no cardinality. - Skip the limit push down when a non-partition filter or a row range index is present: a split's metadata row count is then only an upper bound of what the read returns, so pruning on it could return fewer rows than the limit. - Apply the row ranges selection and the row range groups' deletion vectors to the blob view pre-read, so a reference held only by a dropped row is no longer resolved. Scope: Paimon C++ **reads but does not write** these deletion vectors — the deletes are issued by another engine. `deletion-vectors.bitmap64` and compaction of such a table remain unsupported. Behaviour worth calling out for reviewers: `ApplyPushDownLimit` used to abandon the push down entirely when a split had no countable row count, and now skips that split instead. This affects append and primary key tables too, and has no test, because a deletion file always carries its cardinality in practice. ### Tests Unit tests: | Case | Covers | | --- | --- | | `DataEvolutionUtilsTest.TestRetrieveAnchorFileSkipsBlobAndVectorStoreFiles` | anchor skips blob / vector-store files | | `DataEvolutionUtilsTest.TestRetrieveAnchorFileFailsWithoutNormalFile` | group with no normal file errors | | `DataEvolutionUtilsTest.TestRetrieveAnchorFileTieBreaksWithFileName` | `(max_sequence_number, file_name)` tie-break | | `DataEvolutionSplitReadTest.TestReadGroupDeletionVector` | anchor lookup; null factory, absent and empty vector all yield nullopt | | `DataEvolutionSplitReadTest.TestCreateGroupDvFactoryShiftsPositions` | per-file offset shift, window limit, read-only guards | | `DataEvolutionSplitReadTest.TestCreateGroupDvFactoryRejectsFileOutsideAnchorRange` | file outside the anchor range is rejected | | `DataEvolutionSplitReadTest.TestExcludeDeletedRowIds` | gap range arithmetic, including a fully deleted range | | `DataSplitTest.TestDataEvolutionMergedRowCountSubtractsDeletionFileCardinality` | cardinality subtracted | | `DataSplitTest.TestDataEvolutionMergedRowCountUnavailableWithoutCardinality` | missing cardinality makes the count unavailable | | `DataSplitTest.TestSerializeDataEvolutionSplitWithDeletionFiles` | first row ids + deletion files survive a serialization round trip | | `SchemaValidationTest` (2 cases updated) | the combination is now accepted, including on a blob table | Integration tests (`DataEvolutionTableTest`, parquet/orc/avro): `TestReadWithDeletionVectors`, `TestReadWithDeletionVectorsAcrossReadBatches`, `TestReadWithDeletionVectorsOnPartOfRowRangeGroups`, `TestReadWithDeletionVectorsOnEveryRowRangeGroup` (two groups with their own vectors in one split, plus plan-level assertions on which data file carries which deletion file), `TestReadWithFullyDeletedRowRangeGroup`, `TestReadAfterUpdatingDeletionVectors`, `TestReadWithDeletionVectorsAfterAddingColumn`, `TestLimitPushDownWithHeavilyDeletedFirstRowRangeGroup`, `TestLimitPushDownDisabledByNonPartitionFilter`, `TestLimitPushDownKeptByPartitionFilter`, `TestLimitPushDownDisabledByRowRangeIndex`. Integration tests (`BlobTableInteTest`): `TestDataEvolutionBlobPartialUpdateWithDeletionVectors` (multi-layer blob fallback, placeholder gaps, blob-only projection), `TestBlobViewSkipsDanglingReferenceOfDeletedRow`, `TestBlobViewSkipsDanglingReferenceInEveryRowRangeGroup`, `TestBlobViewPreReadHonorsRowRangesNotPredicate`. Deletion vectors are committed the way an external engine does, through the new `DeletionVectorTestHelper`: a deletion vector index file written with the production `DeletionVectorsIndexFile` writer and committed as an index-only commit message keyed by the group's anchor file. > [!IMPORTANT] > These tests have **not been built or run** yet. The change was prepared without a compile step, so the branch needs a local `unittest` / `data_evolution_table_test` / `blob_table_inte_test` run before merging. The most likely breakage points are the `"Ambiguous table path"` assertions in the blob view tests, whose exact message was inferred from an existing test with the same setup, and the plan-level assertions in `TestReadWithDeletionVectorsOnEveryRowRangeGroup`. ### API and Format No public API under `include/` changes, and no storage format or protocol changes. The deletion vector index format read here is the existing one; this change only makes it reachable for data-evolution tables. `DataSplitImpl::data_deletion_files_` gains a documented invariant (aligned one-to-one with the data files; only anchor files carry an entry) but its serialized form is unchanged, which `DataSplitTest.TestSerializeDataEvolutionSplitWithDeletionFiles` pins. ### Documentation Yes. `docs/source/user_guide/compaction.rst` gains a note stating that a data-evolution table may enable deletion vectors, that Paimon C++ reads but does not write them, that only the default 32-bit vectors can be read, and that compacting such a table is supported by Paimon Java but not ported here. ### Generative AI tooling Generated-by: Claude Opus 5 (1M context) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
