> Well, Parquet is not a single-purpose format. > > Vector stores can choose whatever policy fits their usage, but we're > talking about a general-purpose file format that aims to be broadly > applicable.
What is the gate for Parquet to define something as a logical type? Regarding the current proposal, would it be acceptable if the finiteness requirement was removed? Question for everyone! > Why not? Why shouldn't it be used for storing, for example, NumPy tensor > data (which can contain NaNs and infinites)? A multidimensional arrays logical type might come with other requirements (shapes, strides, dimension names), that vector applications might not have. We might have to make the common general type complex or not rich enough to express everything we need. > It does not make sense to have *both* a "specialist" Vector type and a > "general-purpose" FixedSizeList type. Parquet types are a > general-purpose vocabulary from which you can build up more specialized > applications. Specialist types can be left to domain-specific SW users > of Parquet. > > (for example, Arrow implementations serialize the Arrow schema in > Parquet metadata so that the Arrow schema can be rebuilt at read time, > without mandating that every Arrow datatype has a Parquet equivalent) That does seem to work nicely for Arrow [0]. Should we revisit the Parquet extensions discussion [1][2]? [0] https://arrow.apache.org/docs/format/CanonicalExtensions.html#official-list [1] https://lists.apache.org/thread/9xo3mp4n23p8psmrhso5t9q899vxwfjt [2] https://github.com/apache/parquet-format/pull/451 Best, Rok
