SteNicholas opened a new issue, #169: URL: https://github.com/apache/paimon-cpp/issues/169
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar. ### Motivation Java Paimon supports deletion vectors on data-evolution tables since [`[core] Introducing DeletionVector mechanism for DataEvolution tables` (apache/paimon#8380)](https://github.com/apache/paimon/pull/8380): the `data-evolution.enabled` × `deletion-vectors.enabled` mutual exclusion was dropped from `SchemaValidation.validateRowTracking` (data evolution now only conflicts with `clustering.incremental`), and the scan, read and compaction paths learned to apply deletion files to data-evolution splits. paimon-cpp still carries the old constraint, both in schema validation and in the read path: - `SchemaValidation::ValidateRowTracking` rejects the combination outright (`src/paimon/core/schema/schema_validation.cpp:477-483`). - The same function requires `data-evolution.enabled` for any table holding a BLOB column (`src/paimon/core/schema/schema_validation.cpp:501-504`), so for BLOB tables the two checks form an unsatisfiable pair — a BLOB table can never enable deletion vectors. - Even with the schema check relaxed, reading would still fail: `DataEvolutionSplitRead::ApplyIndexAndDvReaderIfNeeded` returns `Invalid: DataEvolutionSplitRead do not support deletion vector` (`src/paimon/core/operation/data_evolution_split_read.cpp:334-336`), as documented on `src/paimon/core/operation/data_evolution_split_read.h:68`. Creating a table with an INT column and a BLOB column plus `row-tracking.enabled=true`, `data-evolution.enabled=true` and `deletion-vectors.enabled=true` therefore fails at create time with: ```text Invalid: Data evolution config must disabled with deletion-vectors.enabled ``` The current behavior is pinned by `SchemaValidationTest.TestRowTracking` (`src/paimon/core/schema/schema_validation_test.cpp:773-782`). This blocks cross-language use of BLOB + deletion vector tables: a table created and written by a Java engine that includes apache/paimon#8380 can neither be created by nor read through paimon-cpp. It is also the follow-up already noted in #452 ("Java applies deletion vectors to placeholder gap readers; the C++ data-evolution blob read path does not wire deletion vectors yet (pre-existing)"). ### Solution Port the Java design of apache/paimon#8380: 1. **Schema validation** — drop the `data-evolution.enabled` × `deletion-vectors.enabled` check in `SchemaValidation::ValidateRowTracking` and update `SchemaValidationTest.TestRowTracking` accordingly, keeping the other data-evolution constraints (data evolution requires row tracking, a BLOB table must have other normal columns, a BLOB column cannot be a partition key). 2. **Scan and split plumbing** — populate the deletion files of data-evolution splits so every field bunch knows which deletion file applies to which file. `DataSplitImpl` already carries `DeletionFiles()`; `DataEvolutionFileStoreScan` and the data-evolution split generation need to fill and align them, mirroring the Java changes in `DataEvolutionFileStoreScan`, `DataSplit` and the new `DataEvolutionUtils`. 3. **Read path** — wire the existing `ApplyDeletionVectorBatchReader` (`src/paimon/core/deletionvectors/apply_deletion_vector_batch_reader.h`) into `DataEvolutionSplitRead::ApplyIndexAndDvReaderIfNeeded` instead of returning `Invalid`, composing it with the row-range selection and with the blob layers of `BlobFallbackBatchReader`, mirroring Java's `ApplyDeletionFileRecordIterator`, `BlobFallbackRecordReader` and the new `AllPlaceholdersRecordReader`. 4. **Write path** — data-evolution tables are unaware-bucket (`bucket = -1`) append tables, while paimon-cpp only has `BucketedDvMaintainer` (`src/paimon/core/deletionvectors/bucketed_dv_maintainer.h`); producing and merging deletion files for these tables needs the equivalent of Java's `AppendDeleteFileMaintainer`. This step is only required for deleting from C++ — reading a Java-written DV table needs steps 1–3 only. Coverage: schema validation tests for the accepted combination, `DataEvolutionSplitRead` tests for a data-evolution split whose files carry deletion files (including a blob bunch with multiple sequence layers), and a cross-language check that Java and C++ agree on the rows surviving the deletion vector of a BLOB table with row tracking, data evolution and deletion vectors enabled. ### Anything else? apache/paimon#8380 also updated data-evolution compaction (`DataEvolutionCompactCoordinator`, `DataEvolutionCompactTask`) to account for deletion files. paimon-cpp has no data-evolution compaction yet, so that part maps to a follow-up rather than to this issue. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
