JunRuiLee opened a new pull request, #411:
URL: https://github.com/apache/paimon-rust/pull/411
## Purpose
This is **PR 1** of the incremental effort to support reading Paimon
`VECTOR` columns in paimon-rust, tracked in #410.
Apache Paimon has a first-class `VECTOR` type (`VECTOR<element, length>`),
and tables written by Java/Python Paimon can contain `VECTOR` columns.
paimon-rust has no `VectorType` today, so it cannot represent these schemas.
This PR adds the type-system foundation; reading vector *data* is intentionally
deferred to later PRs.
## Scope of this PR
Per the roadmap in #410:
- **PR 1 (this PR)** — Add native `VectorType` to the type system (enum
variant, validation, JSON serde, `Display`).
- PR 2 — Arrow conversion (`VectorType` ↔ `FixedSizeList`) + read vector
columns inlined in ordinary data files.
- PR 3 — Read dedicated `.vector.` files (parquet/vortex).
## What's included
- New `VectorType` mirroring upstream `org.apache.paimon.types.VectorType`:
a fixed-size dense vector with a `u32` length and a restricted element type.
- Validation: length in `[1, i32::MAX]`; element type restricted to
`BOOLEAN`, `TINYINT`, `SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`. Invalid
input returns `DataTypeInvalid`.
- JSON serde matching the Java wire shape: `{"type":"VECTOR"|"VECTOR NOT
NULL","element":<type>,"length":N}` (atomic element types serialize as bare
strings, consistent with the existing `ARRAY` form).
- `Display`: `VECTOR<FLOAT, 128>` with a ` NOT NULL` suffix when not
nullable.
- New `DataType::Vector` enum variant, threaded through every exhaustive
match:
- Type-system semantics implemented: `is_nullable`, `copy_with_nullable`,
`contains_row_type`.
- IO/integration sites return an explicit `Unsupported` error rather than
silently mishandling vectors — actual vector data IO arrives in PR 2/3.
- BTree key comparator rejects `VECTOR` explicitly instead of falling back
to lexicographic byte comparison.
## Behavior after this PR
A schema containing a `VECTOR` column deserializes, validates, serializes
back, and renders correctly. Reading or writing vector *data* returns
`Unsupported` — that is the intended boundary for PR 1.
## Testing
- `cargo test -p paimon --lib` — 895 passed, 0 failed.
- `cargo clippy -p paimon --all-targets` — clean.
- `cargo check -p paimon-datafusion` / `cargo check -p paimon-c` — pass.
Unit tests cover construction/validation (valid + invalid element types,
length bounds including the `i32::MAX` ceiling), exact-JSON serde round-trips
for nullable and `NOT NULL` forms, deserialize rejection of invalid input,
`Display`, untagged-enum dispatch (`VECTOR` vs `ARRAY`), and the `Unsupported`
boundary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
