Thanks for that thought Will! > So is Parquet just growing a vector column type for the fixed-size > embeddings used in nearest-neighbour search to keep up with mainstream > databases, or is it jumping to support the broader set of ML tensor data to > accommodate all the random things an ML engineer might want to jam into a > Parquet file today? I guess that is the question?
Could we have both. The original intent of the design proposal was to cover a general fixed size list that could accommodate various multidimensional array applications (e.g. [1][2]). It seems we would likely want a breaking change for this. This is an effort I want to continue working on. However, there is a lot of immediate interest in resolving the limited vector use case, and that without breaking changes. I proposed specifying the limited vector first to decouple the discussion and design of the general fixed size list from the limited vector type requirements. However this might not be a strong enough motivation to introduce a new logical type. [1] https://arrow.apache.org/docs/format/CanonicalExtensions.html#fixed-shape-tensor [2] https://arrow.apache.org/docs/format/CanonicalExtensions.html#variable-shape-tensor [3] https://github.com/apache/parquet-format/pull/451 Best, Rok
