FANNG1 opened a new issue, #13337:
URL: https://github.com/apache/gravitino/issues/13337
### Describe the feature
Gravitino's type system (`Type.Name`) has no vector type today. A Lance
vector column — Arrow `FixedSizeList<float32, N>` — is not matched by
`LanceDataTypeConverter.toGravitino(Field)`, so it falls through to the
catch-all branch and becomes an opaque `Types.ExternalType` holding the
serialized Arrow field JSON:
```
External("{\"name\":\"embedding\",\"nullable\":true,
\"type\":{\"name\":\"fixedsizelist\",\"listSize\":768},
\"children\":[{\"name\":\"element\",...,\"type\":{\"name\":\"floatingpoint\",\"precision\":\"SINGLE\"},...}]}")
```
This issue proposes a first-class vector type, e.g.
`Types.VectorType.of(elementType, dimension)`, so vector columns are modeled
rather than passed through as strings.
### Motivation
1. **The dimension is invisible to Gravitino.**
`LanceTableOperations#getIndexParamsByIndexType` has to hardcode
`numberOfDimensions = 3` when building an IVF_PQ index, with a TODO, precisely
because the column type carries no dimension it can read.
2. **No validation.** Nothing can check that a `VECTOR` index targets a
vector column, or that an added column's dimension matches what the index
expects.
3. **Not portable across engines.** `ExternalType` is Lance/Arrow-specific
JSON, so the same logical embedding column cannot be mapped to other engines
(e.g. Doris `ARRAY<FLOAT>` with an ANN index — `Index.IndexType.VECTOR` already
exists and maps to `USING ANN`).
4. **Poor UX.** Users of the REST API, Java/Python clients and the web UI
see a raw Arrow JSON blob instead of a readable type.
### Describe the solution
1. Add `VECTOR` to `Type.Name` and a `Types.VectorType` carrying element
type and dimension (a fixed-length list, distinct from the variable-length
`Types.ListType`).
2. Add JSON SerDe support in `common` and a `vector(float32, 768)`-style
parseable literal.
3. Map it in `LanceDataTypeConverter` in both directions: `FixedSizeList`
<-> `VectorType`, and render it as `fixed_size_list` with `length` in
`JsonArrowSchemaConverter` for the Lance REST response.
4. Use the dimension from the column type in
`LanceTableOperations#getIndexParamsByIndexType`, removing the hardcoded value.
5. Extend the Python client and the web UI type mappings.
### Additional context
- Backward compatibility: existing Lance tables materialize their types from
the Arrow schema on every `loadTable`, so they will pick up the new type
automatically; the `ExternalType` form should still be accepted on write.
- Out of scope for this issue, but adjacent: Lance blob columns (Arrow
`LargeBinary` + `lance-encoding:blob=true` field metadata) also fall through to
`ExternalType` today.
- Related: `Index.IndexType.VECTOR` already exists in the API (Doris `USING
ANN`, Lance IVF_PQ).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
