To be clear, this is not an "ease of implementation" concern" (of course
scanning for non-finite values is easy). It's a concern about messaging
consistency and managing expectations: there is no reason for different
logical types to have different validation behaviors in writers. If we
validate FINITE_VECTOR when writing, then the logical expectation is
that we also validate STRING, JSON, etc. when writing.
It is ok for writers to validate data before writing it, but that should
remain an implementation choice.
If we want to take the stance that Parquet implementations should make
it impossible to write invalid Parquet files, then it's a larger change
in the Parquet specification philosophy that needs to be discussed and
decided separately.
As it stands, Parquet C++ certainly offers interesting ways to write
invalid Parquet files.
Regards
Antoine.
Le 04/11/2026 à 09:33, Antoine Pitrou a écrit :
I'm sorry, but I'm -1 (binding) on this.
I don't agree with the requirement that "Writers must enforce these
rules". We don't have any such requirement for STRING, where we simply
state that "The byte array *should* be interpreted as a UTF-8 encoded
character string" (note the deliberately impersonal wording: this does
not impose any constraint on the writer).
In Parquet C++ we don't actively UTF8-validate STRING data before
writing it to Parquet. This responsibility falls on the caller of
Parquet C++ APIs, not on Parquet C++ itself.
(similarly, when creating a UTF8 array in Arrow C++, Arrow C++ does not
actively UTF8-validate the data - though it provides a separate API to
do that explicitly if desired)
I don't understand why Parquet C++ should scan FINITE_VECTOR for NaNs
and Infs before writing them.
Remove the "Writers must enforce these rules" requirement and I'll be +1
on this.
Regards
Antoine.
Le 04/11/2026 à 01:00, Rok Mihevc a écrit :
Hi all,
I'd like to start a vote to add the FINITE_VECTOR logical type to the
Parquet format, as specified in parquet-format PR 624 [1] (at commit
0596fde).
If you'd like to discuss this proposal please use the active discussion
thread [2].
FINITE_VECTOR is roughly:
- annotates the same structure used for LIST, giving unaware readers a
fallback
- every non-null vector has exactly num_elements elements
- elements are required (non-null) and finite (no NaN, null, +Inf or -Inf)
Writers must enforce these rules, and readers may rely on them. This
proposal does not add a new physical representation and doesn't preclude
one later.
This came out of active discussions on the list [2][3] and the community
syncs. The type was renamed from VECTOR to FINITE_VECTOR to indicate it's
specific requirements and to leave room for a more general vector type.
Implementations:
- parquet-java: [4]
- parquet-cpp: [5]
This vote will be open for at least 72 hours.
[ ] +1: Add FINITE_VECTOR to the Parquet spec
[ ] +0: Not going to block, but prefer ...
[ ] -1: Do not add FINITE_VECTOR to Parquet because ...
Thanks to everyone who took part in the discussions and reviews!
Rok
[1] https://github.com/apache/parquet-format/pull/624
[2] https://lists.apache.org/thread/kgmgn2vmrz49thbfhv8b66lglsyd81xt
[3] https://lists.apache.org/thread/xot5f3ghhtc82n1bf0wdl9zqwlrzqks3
[4] https://github.com/apache/parquet-java/pull/3876
[5] https://github.com/apache/arrow/pull/55348
