ChaomingZhangCN opened a new pull request, #295:
URL: https://github.com/apache/paimon-cpp/pull/295
### Purpose
Related issue: #197
Enable VECTOR columns in Parquet-backed, row-tracking append-only tables
with data evolution.
The existing Parquet VECTOR read/write path can support data evolution,
but schema validation currently rejects VECTOR fields in these tables.
This change removes that restriction and adds end-to-end coverage for
full-row writes, partial-column updates, and reads across schema versions.
This PR also:
- Includes VECTOR dimensions in nested type equality checks.
- Rejects incompatible VECTOR dimensions, element types, and conversions
between VECTOR and non-VECTOR types when projecting historical schemas,
including nested fields.
- Replaces two long-suffixed BIGINT literals in the existing Data Evolution
predicate tests with explicit int64_t values to fix macOS linking.
Support remains limited to Parquet. Dedicated vector storage and
schema-commit validation for VECTOR type changes are outside this PR.
### Tests
Added end-to-end tests:
- `TestVectorReadWrite`: full writes, appended row groups, partial VECTOR
updates, NULL transitions, all-NULL updates, and projected row-range reads.
- `TestNestedVectorReadWrite`: full and partial writes of a nullable STRUCT
containing a VECTOR field.
- `TestVectorSchemaEvolution`: adding and dropping VECTOR columns,
null-filling added columns in historical files, and partial writes
across schema versions.
- `TestVectorSchemaEvolutionRejectsTypeChange`: read-time rejection of
incompatible VECTOR dimensions and element types.
Added projection unit tests:
- `PruneDataTypeVector`
- `PruneDataTypeRejectsIncompatibleVectorEvolution`
- `PruneDataTypeRejectsNestedVectorEvolution`
Updated schema-validation coverage to accept VECTOR in Parquet Data
Evolution tables while retaining the ORC restriction.
Local validation on macOS arm64:
- Standard CMake builds of `paimon-core-test` and
`paimon-data-evolution-table-test` succeeded.
- Related core tests: 108 passed.
- Full Data Evolution suite: 204 passed, 18 skipped.
- Related ordinary read/write tests: 33 passed.
- Related BLOB tests: 48 passed.
- cpplint, codespell, sphinx-lint, changed-line clang-format checks,
and `git diff --check` passed.
Runtime results above use statically relinked test executables, without
a Literal compatibility shim. The default mixed-link Data Evolution
executable still aborts during Arrow JSON builder construction.
The full repository suite and Linux CI were not run locally.
New VECTOR integration tests explicitly skip non-Parquet configurations.
### API and Format
No public API, storage format, or protocol changes.
VECTOR values continue to use the existing Parquet LIST representation
and are restored as Arrow FixedSizeList values on read.
Changing a VECTOR's dimension or element type is not supported;
incompatible historical schemas are rejected during read projection.
### Documentation
Updated the VECTOR section of the data types user guide to document
Data Evolution read/write support, adding and dropping VECTOR columns,
and the restrictions on dimension and element-type changes.
### Generative AI tooling
Generated-by: OpenAI Codex CLI 0.153.4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
