rok opened a new pull request, #624: URL: https://github.com/apache/parquet-format/pull/624
### Rationale for this change [Discussion](https://lists.apache.org/thread/kgmgn2vmrz49thbfhv8b66lglsyd81xt) on the [ML](https://lists.apache.org/thread/xot5f3ghhtc82n1bf0wdl9zqwlrzqks3) and led to a decision to split logical annotation proposal from the original proposal, which also aimed to provide a physical representation. This PR proposes the logical annotation only to address the need for a logical vector type. --- This adds a VECTOR logical type stored as a canonical 3-level LIST structure. This is a contract on top of LIST, not a new physical representation: 1. Each non-null vector contains exactly `num_elements` elements. 2. Elements are required to be non-null (`required`) to and finite (no null, Nan, +-Inf) 3. Elements are numeric or boolean Parquet primitives ```xml <vector-repetition> group <name> (VECTOR(<num_elements>)) { repeated group list { required <element-type> element; } } ``` - Element types: BOOLEAN, INT32/INT64 (plain, INT or DECIMAL), FLOAT, DOUBLE, FLOAT16 and FLBA DECIMAL. No nested elements. - Writers must enforce non-null and finite guarantees. Readers may rely on this. - Readers and writers may optimise with this knowledge. - Levels, pages, statistics, bloom filters are as for LIST. Existing encodings continue working. - Writers set ConvertedType.LIST and VECTOR unaware readers read an ordinary LIST. Trade-offs: * Repetition/definition levels are still written. Reader and writer efficiency is left to the implementations. * Adding new float types (e.g. FP8, BF16) requires adding a primitive Parquet type. Packed `FLBA(num_elements * type_width)` storage would decouple this, at the cost of element-level encodings. Out of scope: * Vector indexes, quantisation and normalisation guarantees. Those need their own annotations rather than parameters on VectorType. * A more efficient physical type for storing fixed-sized lists may be proposed in the future and this logical type may be able to annotate it. This is intentionally left out of this proposal. ### What changes are included in this PR? Specifies `VECTOR` logical type, its LIST representation, writer/reader requirements and element-level requirements. ### Do these changes have PoC implementations? Not yet. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
