ChaomingZhangCN opened a new pull request, #268:
URL: https://github.com/apache/paimon-cpp/pull/268
### Purpose
Linked issue: #197
Support VECTOR as an ordinary value column in Parquet-backed deduplicate
primary-key tables.
This change:
- Adds Arrow `FixedSizeList` support to merge-tree row conversion, memory
estimation, spill, and compaction paths.
- Supports VECTOR insert, same-key update, read, projection, and null
value transitions.
- Supports forced spill with updates to the same key across multiple spill
runs.
- Supports adding or dropping VECTOR columns and reading files written
with older schemas.
- Rejects incompatible VECTOR dimension and element-type changes.
- Rejects VECTOR as a primary key, partition key, bucket key, sequence
field, sequence-group ordering field, or sorting field.
- Rejects VECTOR with lookup, partial-update, aggregation, first-row, and
other unsupported merge modes.
- Keeps primary-key tables incompatible with row-tracking Data Evolution.
- Limits the initial primary-key table support to Parquet.
### Tests
Added or extended unit and integration coverage for:
- Non-null and null `FixedSizeList` row conversion.
- Invalid VECTOR dimensions.
- Merge-tree memory estimation and spill.
- Primary-key schema validation for supported and unsupported VECTOR
configurations.
- Deduplicate insert, same-key update, projection, and read.
- VECTOR-to-null and null-to-VECTOR updates.
- Same-key updates across multiple spill runs.
- Full compaction with VECTOR values.
- Add/drop VECTOR schema evolution, including field-ID reordering.
- Incompatible VECTOR dimension and element-type evolution.
- Existing row-tracking Data Evolution VECTOR behavior.
Validation completed:
- Focused FixedSizeList, schema validation, primary-key read/write, spill,
compaction, schema-evolution, and Data Evolution tests passed.
- Full `unittest` target passed: 28/28 test suites.
- `clang-format` checks passed.
- `git diff --check` passed.
### API and Format
No public API under `include/` is changed.
No new storage format or protocol is introduced. VECTOR remains
Parquet-only and continues using the existing Parquet LIST representation,
restored as Arrow
`FixedSizeList` in memory.
### Documentation
Updated the VECTOR support matrix and primary-key table documentation to
describe:
- Deduplicate-only primary-key table support.
- VECTOR value-column restrictions.
- Unsupported key, ordering, lookup, and merge-engine configurations.
- Parquet-only scope.
### Generative AI tooling
Generated-by: OpenAI Codex CLI 0.151.0 (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
