rok opened a new pull request, #55348:
URL: https://github.com/apache/arrow/pull/55348
### Rationale for this change
apache/parquet-format#624 proposes a `VECTOR(num_elements)` logical type. It
annotates a canonical 3-level LIST whose non-null values each contain exactly
`num_elements` required, finite primitive elements. Storage is an ordinary
LIST, so encodings and statistics don't change, and writers also set
`ConvertedType.LIST`. parquet-java has a matching implementation in
apache/parquet-java#3876. Arrow's `FixedSizeList` maps onto this naturally, and
an Arrow implementation also helps validate the proposal.
### What changes are included in this PR?
- `parquet.thrift`: add `VectorType` / `20: VECTOR`, and regenerate with
thrift 0.23.0 (the same version as the checked-in files). Most of the generated
diff is thrift renumbering its temporary variables.
- `LogicalType::Vector(n)` / `VectorLogicalType`: maps to
`ConvertedType::LIST`, has undefined sort order, and requires `n > 0`.
- Creating a VECTOR-annotated `GroupNode` now checks the canonical structure:
- the group is not repeated
- a single unannotated `repeated group list` containing a single `required
element`
- the element is BOOLEAN, FLOAT or DOUBLE, or INT32/INT64 (plain, INT or
DECIMAL), or FLBA (FLOAT16 or DECIMAL)
This check applies when reading files and when building schemas.
`parquet::schema::IsValidVectorElement()` is exposed for reuse.
- Read: a `VECTOR(n)` group is read as `fixed_size_list<element: T not
null>[n]` using the existing `FixedSizeListReader`, which already rejects lists
of the wrong size. If the stored Arrow schema had a nullable element field, it
is restored.
- Write: new experimental
`ArrowWriterProperties::Builder::enable_vector_logical_type()`, off by default.
- With it on, an eligible `FixedSizeList` is written as VECTOR with a
required element. Ineligible ones (string, temporal or nested elements, size 0,
dictionary values) are still written as LIST.
- A null or NaN/±Inf element inside a non-null vector makes the write fail
with `Invalid`. Values under null vectors or null ancestors aren't written, so
they aren't checked.
- The writer follows the schema it derived from the Parquet schema, so
levels and the VECTOR annotation always match.
- pyarrow: `use_vector_logical_type` option on `write_table`,
`ParquetWriter` and `ParquetFileFormat.make_write_options`.
- Docs: add VECTOR to the C++ Parquet logical type table.
### Are these changes tested?
Yes.
- `schema_test.cc`:
- logical type properties, string/JSON representations and sort order
- valid and invalid structures
- thrift round-trip, including invalid `num_elements` and invalid
structure coming from a file
- `arrow_schema_test.cc`:
- Parquet→Arrow mapping, including VECTOR nested in a LIST
- Arrow→Parquet mapping for each allowed element type with nullable and
non-nullable element fields
- disabled by default
- fallback to LIST
- `arrow_reader_writer_test.cc`:
- round-trips with nulls, nested in struct, list and map, and FSL<FSL>
- float16
- null and non-finite element errors
- values under null parents
- reading a hand-written VECTOR(3) file that contains a 2-element list
- `test_data_types.py`:
- round-trips for each element type, with and without a stored schema
- nested types and `fixed_shape_tensor`
- fallback to LIST
- write errors
- the dataset write option
### Are there any user-facing changes?
Yes: a new opt-in writer option in C++ (`enable_vector_logical_type()`) and
Python (`use_vector_logical_type`), marked experimental because the format
proposal isn't accepted yet. Files containing VECTOR are now read as
`FixedSizeList`. Default behaviour doesn't change.
Note for reviewers: released Arrow versions fail to open files with a VECTOR
column ("Logical type Undefined cannot be applied to group node") rather than
falling back to `ConvertedType.LIST`. That's a separate forward-compatibility
issue with unknown group logical types; I'll open it separately.
### Was AI used for this PR?
In accordance to the [AI generation
guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code),
please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [ ] Human
- [x] AI
**Reviewed before submission by:**
- [ ] Human
- [ ] AI
- [ ] Not reviewed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
