Le 23/09/2026 à 12:09, Rok Mihevc a écrit :
The two last items are really gratuitous limitations. Readers are free
to implement limitations of their own and refuse to read more general
fixed-size-list data, but it does not make sense to impose those
limitations in the spec.
The reason for proposing limitations is that this proposal aims for a
narrow vector use case (vector databases, embeddings).
In that context
non-finite or nested values are invalid. Vector stores typically reject such
values.
Well, Parquet is not a single-purpose format.
Vector stores can choose whatever policy fits their usage, but we're
talking about a general-purpose file format that aims to be broadly
applicable.
I don't expect applications that need multidimensional array data to use
this vector logical type
Why not? Why shouldn't it be used for storing, for example, NumPy tensor
data (which can contain NaNs and infinites)?
For general fixed-size-list data, I expect we will continue the discussion
on new physical layout possibilities and provide a general solution there.
Ok, so let's please focus on the general fixed-size-list case then?
It does not make sense to have *both* a "specialist" Vector type and a
"general-purpose" FixedSizeList type. Parquet types are a
general-purpose vocabulary from which you can build up more specialized
applications. Specialist types can be left to domain-specific SW users
of Parquet.
(for example, Arrow implementations serialize the Arrow schema in
Parquet metadata so that the Arrow schema can be rebuilt at read time,
without mandating that every Arrow datatype has a Parquet equivalent)
I don't think it's too unusual for a narrow logical type to restrict its
values.
For instance, JSON requires valid JSON, GEOGRAPHY bounds longitudes
and latitudes.
"JSON requires valid JSON" is tautological, so I don't think this is a
valid analogy.
If we are concerned that writers would have to validate their
data, we could add an explicit property `optional bool VectorType.finite`.
Why would this be be a property of the Vector type, rather than either
1) a piece of statistics or 2) a property of the underlying value type?
Regards
Antoine.
