Thanks for raising the concern and elaborating your point, Antoine! Given the scope of the change you're asking for, it feels like we could iterate on the PR and return to the vote (or call a new one) once we reach consensus? Could you please propose a change on the PR itself: https://github.com/apache/parquet-format/pull/624 ?
If we need a wider discussion, let's use the other thread https://lists.apache.org/thread/kgmgn2vmrz49thbfhv8b66lglsyd81xt . Best, Rok On Wed, Nov 4, 2026 at 9:48 AM Antoine Pitrou <[email protected]> wrote: > > To be clear, this is not an "ease of implementation" concern" (of course > scanning for non-finite values is easy). It's a concern about messaging > consistency and managing expectations: there is no reason for different > logical types to have different validation behaviors in writers. If we > validate FINITE_VECTOR when writing, then the logical expectation is > that we also validate STRING, JSON, etc. when writing. > > It is ok for writers to validate data before writing it, but that should > remain an implementation choice. > > > If we want to take the stance that Parquet implementations should make > it impossible to write invalid Parquet files, then it's a larger change > in the Parquet specification philosophy that needs to be discussed and > decided separately. > > As it stands, Parquet C++ certainly offers interesting ways to write > invalid Parquet files. > > Regards > > Antoine. > > > Le 04/11/2026 à 09:33, Antoine Pitrou a écrit : > > > > I'm sorry, but I'm -1 (binding) on this. > > > > I don't agree with the requirement that "Writers must enforce these > > rules". We don't have any such requirement for STRING, where we simply > > state that "The byte array *should* be interpreted as a UTF-8 encoded > > character string" (note the deliberately impersonal wording: this does > > not impose any constraint on the writer). > > > > In Parquet C++ we don't actively UTF8-validate STRING data before > > writing it to Parquet. This responsibility falls on the caller of > > Parquet C++ APIs, not on Parquet C++ itself. > > > > (similarly, when creating a UTF8 array in Arrow C++, Arrow C++ does not > > actively UTF8-validate the data - though it provides a separate API to > > do that explicitly if desired) > > > > I don't understand why Parquet C++ should scan FINITE_VECTOR for NaNs > > and Infs before writing them. > > > > Remove the "Writers must enforce these rules" requirement and I'll be +1 > > on this. > > > > Regards > > > > Antoine. > > > > > > Le 04/11/2026 à 01:00, Rok Mihevc a écrit : > >> Hi all, > >> > >> I'd like to start a vote to add the FINITE_VECTOR logical type to the > >> Parquet format, as specified in parquet-format PR 624 [1] (at commit > >> 0596fde). > >> If you'd like to discuss this proposal please use the active discussion > >> thread [2]. > >> > >> FINITE_VECTOR is roughly: > >> - annotates the same structure used for LIST, giving unaware readers a > >> fallback > >> - every non-null vector has exactly num_elements elements > >> - elements are required (non-null) and finite (no NaN, null, +Inf or > -Inf) > >> > >> Writers must enforce these rules, and readers may rely on them. This > >> proposal does not add a new physical representation and doesn't preclude > >> one later. > >> > >> This came out of active discussions on the list [2][3] and the community > >> syncs. The type was renamed from VECTOR to FINITE_VECTOR to indicate > it's > >> specific requirements and to leave room for a more general vector type. > >> > >> Implementations: > >> - parquet-java: [4] > >> - parquet-cpp: [5] > >> > >> This vote will be open for at least 72 hours. > >> > >> [ ] +1: Add FINITE_VECTOR to the Parquet spec > >> [ ] +0: Not going to block, but prefer ... > >> [ ] -1: Do not add FINITE_VECTOR to Parquet because ... > >> > >> Thanks to everyone who took part in the discussions and reviews! > >> > >> Rok > >> > >> [1] https://github.com/apache/parquet-format/pull/624 > >> [2] https://lists.apache.org/thread/kgmgn2vmrz49thbfhv8b66lglsyd81xt > >> [3] https://lists.apache.org/thread/xot5f3ghhtc82n1bf0wdl9zqwlrzqks3 > >> [4] https://github.com/apache/parquet-java/pull/3876 > >> [5] https://github.com/apache/arrow/pull/55348 > > > > >
