I think separating them into FINITE_VECTOR and FIXED_LENGTH_LIST is a good idea, though I think renaming FINITE_VECTOR to VECTOR looks more natural. Should we move forward with proposing FINITE_VECTOR/VECTOR first?
On Tue, Oct 13, 2026 at 6:26 PM Alkis Evlogimenos via dev < [email protected]> wrote: > I suggest we go with VECTOR for the finite vector to keep the name the same > as other engines/type systems. If we want a non-finite one in the future we > can name that something else or we add fixed size list that will serve the > same purpose without a special name. > > > On Thu, Oct 8, 2026 at 10:20 PM Ryan Blue <[email protected]> wrote: > > > A logical type would: > > - Describe data for users and readers to understand what the file > contains > > - Signal that repetition and definition levels will reflect a fixed > number > > of elements, which implementations can use to optimize decoding > > - Guarantee that there are no null elements (because the type cannot be > > applied to a list with optional elements) > > - [FINITE_VECTOR] Guarantee that no elements are NaN or Infinity values > so > > that the vectors have a finite length and are suitable for distance > > calculations > > > > The finite value restriction is a restriction on writers to prevent bad > > data from being written. > > > > Ignoring logical types that are not understood is a rule for readers. A > > reader can handle vectors and pass it on to another layer. For instance, > a > > Parquet reader that doesn't support geometry passes values as `binary`. > > That binary value can be passed to a geometry UDF, even if the underlying > > engine and Parquet reader doesn't have support. > > > > A tool that doesn't understand FINITE_VECTOR can similarly pass a list of > > floats along to a UDF. But it is still important to the user that bad > > values are not present in the data. In your example, bad values are > vectors > > that cause the centroid calculation to fail, like vectors with infinite > > length. > > > > Ryan > > > > On Thu, Oct 8, 2026 at 12:54 PM Will Edwards via dev < > > [email protected]> > > wrote: > > > > > Could we clarify the use cases for a VECTOR annotation on LIST<FLOAT>? > > > Specifically, what does it make possible that's impossible today, and > > what > > > does it make easier that's currently hard? > > > > > > I'm struggling to find a strong answer. I'd like every engine to offer > > > vector functions on lists of floats without needing a new logical type. > > > > > > To be clear, I'm still enthusiastic about a vector type that describes > > how > > > to unpack and scale composite representations (quantised codes plus > > scales, > > > codebooks, etc.), in the direction ML is going. That adds information > > > engines don't have. A label on LIST<FLOAT> doesn't: engines already > > > understand the physical type, and many already offer vector functions > on > > > it. I don't see a performance benefit, and NaN/null/Inf checks are > cheap > > > enough that engines should do them anyway. The label also doesn't say > > what > > > you can do with the column. Knowing that a column holds embeddings > > doesn't > > > say how they can be used, just as zipcode and age are both integers but > > > multiplying them is meaningless. > > > > > > My concern is the opposite effect. We're establishing that engines > should > > > ignore logical types they don't understand. So a tool that doesn't know > > > VECTOR will read the column as LIST<FLOAT>, let the user compute > > centroids > > > (say), and write plain LIST<FLOAT> back out. If other engines restrict > > > embedding functions to VECTOR-annotated columns, that file stops > working > > > with them, through no fault of the user. An annotation that doesn't > > change > > > what the data is shouldn't be able to make it unusable. > > > > > > Am I missing a use case? > > > > > > Best, Will > > > > > > On Thu, 8 Oct 2026 at 19:53, Steven Wu <[email protected]> wrote: > > > > > > > I also think a FINITE_VECTOR type brings a lot of values to many use > > > cases. > > > > It is would be great to make progress on the FINITE_VECTOR, and not > to > > > fit > > > > every use cases into the FIXED_LENGTH_LIST. > > > > > > > > On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue <[email protected]> wrote: > > > > > > > > > Going through this thread again, it looks like there are a few > things > > > > that > > > > > people generally agree on: > > > > > > > > > > - In the short term, we want to introduce a logical type or types; > a > > > > better > > > > > physical representation can be done later. We can focus on logical > > > types. > > > > > - There is precedent in logical types for restricting values, like > > UTF8 > > > > > strings to contain only UTF-8 codepoints. I think we also agree > that > > we > > > > > could restrict vector elements to non-nested or primitive types. > > > > > - There are many examples of databases that restrict their vector > > types > > > > to > > > > > non-null, finite elements. > > > > > - Some people want to have write-time guarantees that vectors have > > only > > > > > non-null, finite elements > > > > > - Some people want to extend this work to include tensors, shorter > > > > floating > > > > > point representations, variable lengths, NaN or Infinity elements, > > etc. > > > > > > > > > > It sounds like we are nearly there, but have two halves of the > > > community > > > > > that care about different things. That just means we should have > > > multiple > > > > > types. It's entirely reasonable to want a type that allows NaN > > values, > > > > and > > > > > it's also reasonable for those of us building databases, table > > > standards, > > > > > and vector indexes to care about guarantees that elements are > finite. > > > > > There's more than enough examples to show that this is a legitimate > > use > > > > > case. > > > > > > > > > > Since I don't see the concerns of the different use cases changing, > > it > > > > > looks like we need to find a compromise to move forward. I propose > > that > > > > we > > > > > introduce a FINITE_VECTOR type and separately a FIXED_LENGTH_LIST > or > > > > > another VECTOR type that allows infinite and NaN values. > > > > > > > > > > Can we agree on a compromise? Is there a strong reason to try to > fit > > > > these > > > > > use cases into a single type? What if that means not making > progress? > > > > > > > > > > Ryan > > > > > > > > > > On Mon, Oct 5, 2026 at 12:15 PM Philipp Fischbeck via dev < > > > > > [email protected]> wrote: > > > > > > > > > > > Thinking about this more, I am also fine with pulling out the > > > “NaN/Inf” > > > > > > requirement. Either into a type boolean (my preference), or > putting > > > no > > > > > > restrictions on the Parquet level, allowing all IEEE 754 float > > > numbers. > > > > > If > > > > > > some top-level application wants to put additional constraints on > > the > > > > > data > > > > > > type, like Iceberg not allowing NaN/infinity, it can still do so > > one > > > > > layer > > > > > > above. It seems like this was the biggest point of discussion > > > > remaining. > > > > > > All these options work for me, and we should move forward with > this > > > > > > proposal. > > > > > > > > > > > > Concerning additional element types like bf16, they can be > > introduced > > > > as > > > > > > Parquet types (at least in the current LIST-based spec proposal). > > > > > > > > > > > > Concerning Will’s suggestion around representing VECTOR as a > > variable > > > > > > length list with a MIN/MAX_LENGTH metadata restriction: I think > > that > > > a > > > > > > dedicated logical VECTOR type still provides enough benefits by > > > > > describing > > > > > > the semantics of the data, and allowing to capture future changes > > > > around > > > > > > statistics. In that case, the length is better captured as an > exact > > > > > length > > > > > > for this type specifically. I’m also not sure about other use > cases > > > for > > > > > > MIN_LENGTH, so introducing a new type constraint system just for > > the > > > > > > MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have enough value > in > > > my > > > > > > view. > > > > > > > > > > > > Best, > > > > > > Philipp > > > > > > > > > > > > > > > > > > On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev < > > > > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > +1 Sebastian, a vector type suggests usage beyond normalized > > > > embeddings > > > > > > and > > > > > > > engines really aren't going to be able to elide all the checks > > > > anyway. > > > > > > > > > > > > > > Antoine, you suggested we explore metadata solutions too :) > > > > > > > > > > > > > > Today, a database supporting CHAR(N) or VARCHAR(n) can't > express > > > this > > > > > in > > > > > > > Parquet. Engines can write opaque blobs they understand in the > > > > > metadata > > > > > > > (e.g. Arrow does) so the extra type info round-trips, but this > > > isn't > > > > > > > standardized, portable, nor interoperable. > > > > > > > > > > > > > > So perhaps standardizing constraint metadata or new Thrift > fields > > > (my > > > > > > pref) > > > > > > > can generically support VECTOR(N) whilst also being more > broadly > > > > useful > > > > > > for > > > > > > > other use cases and data types? > > > > > > > > > > > > > > Imagine metadata or Thrift fields to tag the MIN_LENGTH and > > > > MAX_LENGTH > > > > > > for > > > > > > > Parquet's variable-length physcial types. These would count > > > logical > > > > > > units > > > > > > > (e.g. UTF8 code points for STRING) rather than bytes. > > > > > > > > > > > > > > Then one engine's VECTOR(N) can be written to Parquet as > LIST(N) > > > with > > > > > > > MIN_LENGTH = MAX_LENGTH = N and then readers - even different > > > > engines - > > > > > > can > > > > > > > pick up these constraints if they support doing so. We don't > > > > actually > > > > > > need > > > > > > > a logical type to express a VECTOR then. > > > > > > > > > > > > > > This also allows standard SQL's CHAR(N) and VARCHAR(N) to be > > > > expressed > > > > > on > > > > > > > UTF8 columns etc. > > > > > > > > > > > > > > This also seems generally useful in the future as we approach > > > > > expressing > > > > > > > other tensor data shapes and types. > > > > > > > > > > > > > > This won't change the mechanics: the storage type doesn't > enforce > > > > these > > > > > > > constraints and engines must check them. It's previously > > > established > > > > > that > > > > > > > performance is just a software problem and there is no > > fundamental > > > > > > > performance advantage to the fixed size types in Dremel-like > > > formats > > > > > like > > > > > > > Parquet. > > > > > > > > > > > > > > Best, > > > > > > > Will > > > > > > > > > > > > > > On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard < > > > > > > [email protected] > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > After listening in on the last meeting Wednesday, I have been > > > > > thinking > > > > > > > > about this quite a bit, and thought I would share my 2 cents. > > > > > > > > > > > > > > > > I suggest not requiring VECTOR elements to be finite, as > > proposed > > > > in > > > > > > the > > > > > > > PR > > > > > > > > <https://github.com/apache/parquet-format/pull/624>. > > > > > > > > > > > > > > > > While Divjot and others note the specialization for UTF-8, > > > Parquet > > > > > > > already > > > > > > > > defines DOUBLE and FLOAT as IEEE encodings > > > > > > > > < > > > > > > https://github.com/apache/parquet-format/blob/master/README.md#types > > > > > > >. > > > > > > > > > > > > > > > > Making NaN or +/-Inf illegal inside VECTOR would be a new > > > > restriction > > > > > > on > > > > > > > > those same underlying physical types. Scalar FLOAT/DOUBLE > > columns > > > > > have > > > > > > > > never had such a rule. > > > > > > > > > > > > > > > > It would also not match how the systems that actually produce > > > these > > > > > > > vectors > > > > > > > > treat floats. Most of them use IEEE floats, e.g. Spark, > DuckDB, > > > > > NumPy, > > > > > > > > PyTorch, TensorFlow, or lower level CUDA, oneMKL, and > OpenBLAS. > > > > > > > > > > > > > > > > pgvector's "all elements must be finite" rule is an insert > > > policy. > > > > > > While > > > > > > > I > > > > > > > > think Parquet as a general-purpose format should allow them > out > > > of > > > > > the > > > > > > > box. > > > > > > > > It is a user system decision to disallow writing these values > > or > > > to > > > > > > throw > > > > > > > > errors when encountering them. > > > > > > > > > > > > > > > > A type-level ban also does not keep Inf/NaN out of vector > math. > > > > > Finite > > > > > > > > coordinates still overflow. For instance: magnitude = > > > ||[FLOAT_MAX, > > > > > > > > FLOAT_MAX]|| where every coordinate is finite overflows to > > +Inf. > > > > > > PyTorch > > > > > > > > documents the same overflow: tensor([1e20, 1e20]).norm() is > inf > > > in > > > > > > fp32. > > > > > > > > > > > > > > > > A generated MAGNITUDE(vec) column would then be Inf while the > > > > VECTOR > > > > > > > itself > > > > > > > > was required to be finite. The ban only forbids storing > > specials > > > in > > > > > the > > > > > > > > coordinates, and does not prevent even simple downstream > > > > computations > > > > > > > from > > > > > > > > producing them. > > > > > > > > > > > > > > > > I suggest keeping VECTOR as the type and add an optional bool > > > > > > > > VectorType.finite (default false). Engines that want > > > pgvector-like > > > > > > > > rejection can require the flag = true. Engines that use > NumPy / > > > > > > PyTorch / > > > > > > > > Spark arrays can keep representable IEEE values. For the same > > > > > reason, I > > > > > > > do > > > > > > > > not favor adding yet another type called FINITE_VECTOR. > > > > > > > > > > > > > > > > Best, > > > > > > > > Sebastian > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev < > > > > > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > I agree with moving forward with the proposal and renaming > > > VECTOR > > > > > to > > > > > > > > > FINITE_VECTOR or something > > > > > > > > > similar if others feel that better communicates its > intent. I > > > see > > > > > > value > > > > > > > > in > > > > > > > > > having a vector type that guarantees > > > > > > > > > its values are mathematically-correct vectors for a few > > > reasons: > > > > > > > > > > > > > > > > > > 1. As others have mentioned, this is similar to the STRING > > > > logical > > > > > > type > > > > > > > > > mandating UTF-8. > > > > > > > > > 2. It is not possible to rely on statistics for detecting > NaN > > > or > > > > > > +-inf > > > > > > > > > values as all statistics are optional. > > > > > > > > > To my knowledge, there is no precedent for requiring that > > > writers > > > > > > emit > > > > > > > > > stats for specific logical types. > > > > > > > > > 3. Without statistics, NaN/inf values could flow through > the > > > > engine > > > > > > and > > > > > > > > > block them for specific functionality. > > > > > > > > > This seems to be Lance's behavior. However, I don't think > > this > > > is > > > > > as > > > > > > > easy > > > > > > > > > as it sounds and Lance has run > > > > > > > > > into several bugs due to this ([1] [2] for examples). Their > > > > Python > > > > > > [3] > > > > > > > > and > > > > > > > > > Rust [4] APIs default to erroring on > > > > > > > > > vectors that contain NaN values. > > > > > > > > > > > > > > > > > > I agree that Parquet is a general-purpose format, but I > don't > > > > think > > > > > > > this > > > > > > > > > implies the lack of domain-specific types. > > > > > > > > > Instead, this broadens the use cases where Parquet would be > > > > > > considered > > > > > > > a > > > > > > > > > good fit. I also fully agree with > > > > > > > > > continuing work on a general fixed-size list type if we > > believe > > > > > there > > > > > > > are > > > > > > > > > non-vector use cases that would benefit > > > > > > > > > from such guarantees at the storage layer. > > > > > > > > > > > > > > > > > > [1] https://github.com/lance-format/lance/pull/3648 > > > > > > > > > [2] https://github.com/lance-format/lance/issues/8607 (see > > > > point 2 > > > > > > in > > > > > > > > > "root > > > > > > > > > cause") > > > > > > > > > [3] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911 > > > > > > > > > [4] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110 > > > > > > > > > > > > > > > > > > On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via dev < > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > I also favor moving forward with the proposal as is. I > see > > > > strong > > > > > > > value > > > > > > > > > and > > > > > > > > > > use cases for the numeric vector type, and I agree with > the > > > > > > > > restrictions > > > > > > > > > > around NaN/Inf values and element types in general. > > Precedent > > > > > > exists > > > > > > > in > > > > > > > > > > other types that restrict which physical type values are > > > > > logically > > > > > > > > valid > > > > > > > > > > (e.g., the geospatial types needing to be WKB-encoded). > A > > > > > general > > > > > > > > > > fixed-size list also has value, but I think it makes > sense > > to > > > > > > > separate > > > > > > > > > the > > > > > > > > > > two types. The semantics are relevant, and I can see how > > the > > > > two > > > > > > > types > > > > > > > > > > might diverge further in the future regarding statistics > or > > > > > > > encodings. > > > > > > > > > > > > > > > > > > > > I would be okay with renaming the vector type, but it > seems > > > > like > > > > > > the > > > > > > > > > VECTOR > > > > > > > > > > name is already used across other engines mentioned in > this > > > > > thread. > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > Philipp > > > > > > > > > > > > > > > > > > > > On Sat, Sep 26, 2026 at 4:37 PM Gang Wu < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > To separate orthogonal concerns, I feel we should treat > > the > > > > > > general > > > > > > > > > > > FixedSizeList as the underlying physical layout, and > > > separate > > > > > it > > > > > > > from > > > > > > > > > the > > > > > > > > > > > discussion of a specialized VECTOR (or broader ML > tensor) > > > > > logical > > > > > > > > type. > > > > > > > > > > > > > > > > > > > > > > Currently, the immediate need is to define a > specialized > > > > VECTOR > > > > > > > type > > > > > > > > so > > > > > > > > > > > that different engines and formats have a common > semantic > > > > > > contract > > > > > > > > for > > > > > > > > > > > interoperability. > > > > > > > > > > > > > > > > > > > > > > If the main point of contention is just the > NaN/Inf/null > > > > > > > limitations, > > > > > > > > > > maybe > > > > > > > > > > > we can resolve this by adding a qualification to the > name > > > > (like > > > > > > > > Andrew > > > > > > > > > > > suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR) to > > > avoid > > > > > > > > confusion > > > > > > > > > > with > > > > > > > > > > > broader ML use cases. Alternatively, we could > introduce a > > > > > boolean > > > > > > > > flag > > > > > > > > > > > (e.g., optional bool VectorType.finite) as proposed > > > earlier. > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb < > > > > > > > [email protected]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > I am a little confused about the points of > disagreement > > > > here. > > > > > > > > > > > > > > > > > > > > > > > > Is it that the name `VECTOR` is too general for the > > > > specific > > > > > > use > > > > > > > > case > > > > > > > > > > of > > > > > > > > > > > > fixed length arrays with non-finite values used for > > > vector > > > > > > > > > embeddings? > > > > > > > > > > > If > > > > > > > > > > > > so, maybe we could add a qualification to the name > > > > > > > (FINITE_VECTOR? > > > > > > > > > > > > FEATURE_VECTOR? VECTOR_EMBEDDING?) > > > > > > > > > > > > > > > > > > > > > > > > It seems like there is precedent for defining logical > > > types > > > > > > that > > > > > > > > > > specify > > > > > > > > > > > > certain physical types and restrict their contents to > > > help > > > > > > > readers > > > > > > > > > > > > interpret meaning, for example string[1] or JSON; > > > Defining > > > > > one > > > > > > > for > > > > > > > > > > > vector > > > > > > > > > > > > embeddings seems reasonable enough to me. > > > > > > > > > > > > > > > > > > > > > > > > Andrew > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1]: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks < > > > > > > [email protected]> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Well, Parquet is not a single-purpose format. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Vector stores can choose whatever policy fits > their > > > > > usage, > > > > > > > but > > > > > > > > > > we're > > > > > > > > > > > > > > talking about a general-purpose file format that > > aims > > > > to > > > > > be > > > > > > > > > broadly > > > > > > > > > > > > > > applicable. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I agree that the format is intended to be general > > > > purpose, > > > > > > but > > > > > > > > > types > > > > > > > > > > > > within > > > > > > > > > > > > > the format define specific representation and > > semantics > > > > to > > > > > > > > achieve > > > > > > > > > > that > > > > > > > > > > > > > goal. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > "Vector" has a more general sense than ML and > it's > > > > quite > > > > > > > > trivial > > > > > > > > > to > > > > > > > > > > > > > > produce vectors with non-finite values using e.g. > > > NumPy > > > > > or > > > > > > > > Pandas > > > > > > > > > > (or > > > > > > > > > > > > > > any other piece of dataframe or scientific > > computing > > > > > > > software). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The justification for restricting non-finite > values > > is > > > > > > > directly > > > > > > > > > > > related > > > > > > > > > > > > to > > > > > > > > > > > > > the semantic meaning of a vector. Vectors have > > > specific > > > > > > > > > > > characteristics > > > > > > > > > > > > > (dimension, magnitude, direction, etc). Non-finite > > > > values > > > > > > are > > > > > > > > > > invalid > > > > > > > > > > > > and > > > > > > > > > > > > > break those characteristics. I'd liken it to > storing > > > > > > non-utf8 > > > > > > > > > values > > > > > > > > > > > in > > > > > > > > > > > > a > > > > > > > > > > > > > string and still calling it a string. > > > > > > > > > > > > > > > > > > > > > > > > > > That also differentiates a vector from a fixed > length > > > > list, > > > > > > > which > > > > > > > > > is > > > > > > > > > > > more > > > > > > > > > > > > > general and does not carry the same semantic > meaning. > > > > > > > > > > > > > > > > > > > > > > > > > > I also don't think we need to bundle all similarly > > > shaped > > > > > use > > > > > > > > cases > > > > > > > > > > > into > > > > > > > > > > > > > the same solution (e.g. flxed size list). Vector > is > > a > > > > well > > > > > > > > > > established > > > > > > > > > > > > > data type at this point and aligns with the general > > > > purpose > > > > > > > > nature > > > > > > > > > of > > > > > > > > > > > the > > > > > > > > > > > > > format (many other formats and databases natively > > > support > > > > > > > this). > > > > > > > > > > > > > > > > > > > > > > > > > > -Dan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou < > > > > > > > > > [email protected]> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Le 24/09/2026 à 20:11, Russell Spitzer a écrit : > > > > > > > > > > > > > > > I think something I haven't seen yet is the > clear > > > > > > use-case > > > > > > > > for > > > > > > > > > > > > > non-finite > > > > > > > > > > > > > > > values within a vector type. There has been a > lot > > > of > > > > > > > > > theoretical > > > > > > > > > > > > > > discussion > > > > > > > > > > > > > > > here about use cases but what toolings > currently > > > > > produce > > > > > > > > these > > > > > > > > > > > > vectors > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > how do they function in downstream engines? I'd > > > like > > > > to > > > > > > > limit > > > > > > > > > > > > > introducing > > > > > > > > > > > > > > > types that have a wide range of values which > are > > > > simply > > > > > > are > > > > > > > > > > failure > > > > > > > > > > > > > modes > > > > > > > > > > > > > > > of different flavors > > > > > > > > > > > > > > > > > > > > > > > > > > > > "Vector" has a more general sense than ML and > it's > > > > quite > > > > > > > > trivial > > > > > > > > > to > > > > > > > > > > > > > > produce vectors with non-finite values using e.g. > > > NumPy > > > > > or > > > > > > > > Pandas > > > > > > > > > > (or > > > > > > > > > > > > > > any other piece of dataframe or scientific > > computing > > > > > > > software). > > > > > > > > > > > > > > > > > > > > > > > > > > > > It's not like the general concept of a vector is > a > > > IETF > > > > > > > > standard > > > > > > > > > or > > > > > > > > > > > > > > anything, so analogies with JSON or UTF8 (where > the > > > > > values > > > > > > > are > > > > > > > > > > > expected > > > > > > > > > > > > > > to be valid according to the respective specs) do > > not > > > > > > apply. > > > > > > > > > > > > > > > > > > > > > > > > > > > > This discussion would IMHO have a better chance > of > > > > > > converging > > > > > > > > if > > > > > > > > > we > > > > > > > > > > > > > > separated the two orthogonal concerns of: > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. representing fixed-length list/vector data in > > > > Parquet > > > > > > > (with > > > > > > > > a > > > > > > > > > > > > > > potential limitation to non-nested types) > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. representing the information that a given > column > > > (or > > > > > > > column > > > > > > > > > > chunk, > > > > > > > > > > > > > > etc.) does not contain non-finite values > > > > > > > > > > > > > > > > > > > > > > > > > > > > ... both of which are useful on their own right, > > but > > > do > > > > > not > > > > > > > > have > > > > > > > > > to > > > > > > > > > > > be > > > > > > > > > > > > > > conflated and bundled together. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > > > > > > > > > > > > > > Antoine. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
