rok opened a new issue, #55347:
URL: https://github.com/apache/arrow/issues/55347
### Describe the enhancement requested
There is an open proposal to add a `VECTOR` logical type to Parquet
(apache/parquet-format#624). It annotates a canonical 3-level LIST whose
non-null values each have exactly `num_elements` required, finite primitive
elements. The storage is an ordinary LIST, so existing encodings, statistics,
etc. are unchanged. The main point is to let embeddings and similar fixed-size
data say "this is fixed-size" in the file. parquet-java has a matching
implementation in apache/parquet-java#3876.
It would be good to have an Arrow C++ implementation as well, both to help
validate the proposal and because Arrow `FixedSizeList` maps onto it naturally:
- Read: a `VECTOR(n)` group becomes `fixed_size_list<element: T not
null>[n]`. The existing FixedSizeList reader already rejects lists of the wrong
size.
- Write: opt-in via an `ArrowWriterProperties` flag, with
`use_vector_logical_type` in pyarrow, off by default while the spec is a draft.
- A FixedSizeList whose element type maps to BOOLEAN, INT32/INT64 (plain,
INT or DECIMAL), FLOAT, DOUBLE, FLOAT16 or FLBA DECIMAL is written as VECTOR
with a required element. Anything else is written as a plain LIST, as today.
- The writer has to enforce the spec's rules, so a null or NaN/Inf element
inside a non-null vector is an error.
- Schema: validate the canonical structure when building or reading the
schema (`list`/`element` names, required element, allowed element types).
I have a prototype covering the thrift update, the logical type, the schema
mapping, the writer checks and a pyarrow round-trip test. I'll open a PR once
this is filed. Since the spec isn't accepted yet, the PR would be marked
experimental.
### Component(s)
C++, Parquet, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
