JunRuiLee opened a new pull request, #440:
URL: https://github.com/apache/paimon-rust/pull/440
## Summary
Part of #410 (PR 4). Integrates the native `VECTOR<FLOAT, N>` type with the
Lumina
index build path. Previously only `ARRAY<FLOAT>` columns could be indexed;
this
extends validation, dimension resolution, and Arrow vector extraction to also
accept `DataType::Vector`, deriving the dimension from the column type
(`VectorType::length`) instead of requiring an explicit
`lumina.index.dimension`
option. This mirrors Java
`LuminaVectorGlobalIndexWriter.validateAndResolveDimension`,
where both `ARRAY<FLOAT>` and `VECTOR<FLOAT>` are valid vector-index columns.
## Changes
- **Validation** (`validate_vector_field`): accept `VECTOR<FLOAT>` in
addition to
`ARRAY<FLOAT>`; reject non-float element types (e.g. `VECTOR<DOUBLE>`).
- **Dimension resolution** (`effective_lumina_options`, new): for a
`VECTOR<FLOAT, N>` column, derive `lumina.index.dimension = N` from the
type.
Absent → inject `N`; present-and-equal → keep; present-and-different →
`ConfigInvalid`. Non-vector columns (`ARRAY<FLOAT>`) are returned
unchanged so
existing option/default behavior is preserved. Matches Java's
`testVectorTypeRejectsExplicitDimensionConflict`.
- **Extraction** (`extract_vectors_from_batches`): resolve the vector column
as
either `List<Float32>` (ARRAY) or `FixedSizeList<Float32>` (VECTOR) via a
`VectorLayout` enum; both yield a `Float32Array` plus a per-row `[start,
end)`
slice, with identical null-vector / null-element / dimension-mismatch
handling.
- Add `LUMINA_DIMENSION_OPTION` constant in `lumina/mod.rs`.
## Scope note
#410's PR 4 description names `lumina`/`vindex` index build **and**
`VectorSearchBuilder`. Only the Lumina build path is touched here,
deliberately:
- **vindex index build** does not exist in Rust yet — `src/vindex/`
currently has
only a reader (from #399), no writer/builder. VECTOR support for vindex
will be
built in when the vindex build path is ported (the Java vindex writer
already
accepts both `ARRAY<FLOAT>` and `VECTOR<FLOAT>`, so it will be internal to
that
work rather than a follow-up integration).
- **`VectorSearchBuilder`** dispatches by field name and index-file type
against
already-built index files; it does not re-validate the source column as
ARRAY
vs VECTOR, so it is transparent to the column type and needs no change.
## Testing
- [x] `cargo fmt`
- [x] `cargo check -p paimon`
- [x] `cargo test -p paimon --lib lumina_index_build_builder` (28 passed, 1
ignored — the ignored test requires `LUMINA_LIB_PATH`)
New tests cover: VECTOR<FLOAT> accepted / VECTOR<DOUBLE> rejected; dimension
absent-inject / matching-ok / mismatch-errors / array-unchanged; dimension
propagation into native Lumina options; and FixedSizeList<Float32> extraction
incl. null-vector, null-element, and dimension-mismatch rejection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
