I'm sorry, but I'm -1 (binding) on this.
I don't agree with the requirement that "Writers must enforce these
rules". We don't have any such requirement for STRING, where we simply
state that "The byte array *should* be interpreted as a UTF-8 encoded
character string" (note the deliberately impersonal wording: this does
not impose any constraint on the writer).
In Parquet C++ we don't actively UTF8-validate STRING data before
writing it to Parquet. This responsibility falls on the caller of
Parquet C++ APIs, not on Parquet C++ itself.
(similarly, when creating a UTF8 array in Arrow C++, Arrow C++ does not
actively UTF8-validate the data - though it provides a separate API to
do that explicitly if desired)
I don't understand why Parquet C++ should scan FINITE_VECTOR for NaNs
and Infs before writing them.
Remove the "Writers must enforce these rules" requirement and I'll be +1
on this.
Regards
Antoine.
Le 04/11/2026 à 01:00, Rok Mihevc a écrit :
Hi all,
I'd like to start a vote to add the FINITE_VECTOR logical type to the
Parquet format, as specified in parquet-format PR 624 [1] (at commit
0596fde).
If you'd like to discuss this proposal please use the active discussion
thread [2].
FINITE_VECTOR is roughly:
- annotates the same structure used for LIST, giving unaware readers a
fallback
- every non-null vector has exactly num_elements elements
- elements are required (non-null) and finite (no NaN, null, +Inf or -Inf)
Writers must enforce these rules, and readers may rely on them. This
proposal does not add a new physical representation and doesn't preclude
one later.
This came out of active discussions on the list [2][3] and the community
syncs. The type was renamed from VECTOR to FINITE_VECTOR to indicate it's
specific requirements and to leave room for a more general vector type.
Implementations:
- parquet-java: [4]
- parquet-cpp: [5]
This vote will be open for at least 72 hours.
[ ] +1: Add FINITE_VECTOR to the Parquet spec
[ ] +0: Not going to block, but prefer ...
[ ] -1: Do not add FINITE_VECTOR to Parquet because ...
Thanks to everyone who took part in the discussions and reviews!
Rok
[1] https://github.com/apache/parquet-format/pull/624
[2] https://lists.apache.org/thread/kgmgn2vmrz49thbfhv8b66lglsyd81xt
[3] https://lists.apache.org/thread/xot5f3ghhtc82n1bf0wdl9zqwlrzqks3
[4] https://github.com/apache/parquet-java/pull/3876
[5] https://github.com/apache/arrow/pull/55348
