Hi Sagar, thanks for the detailed responses. I’ve gone through the email and will take some time to review everything properly. I’ll get back to you once I’ve had a chance to digest the details.
I’d also suggest joining the community call so we can discuss this with the broader community. You could walk us through the design decisions you’ve made in the repo and capture them in a Google Doc. Once we’ve discussed and aligned on those points, we can move forward with the FIP. Best Regards, Mehul Batra On Sun, Aug 30, 2026 at 9:30 PM Sagar <[email protected]> wrote: > Hi Mehul, > > PFA my responses to the questions: > > *Zero-Copy Scope:* > > Zero-copy is guaranteed for the native columnar tiering pipeline (Fluss > Arrow Log-> Arrow FixedSizeListVector-> Lance Dataset). Row-oriented > writes and Flink SQL query execution follow standard row/array > serialization (InternalArray<-> Flink ArrayData). > > *Dimension as part of Datatype:* > > The POC already implements dimension as a first class parameter of > VectorType, i.e you specify it with Vector(n). > > *Dimension Validation at Write boundary:* > > Validation is enforced at the client write boundary (AppendWriter, > UpsertWriter, BinaryWriter, and Flink Sink serializer) as well as the Arrow > batch writer (ArrowVectorWriter). When writing to a VECTOR(n) column, the > writer verifies array.size() == n before serializing or sending data. > > Failure mode: If an incoming vector has n − 1 or n + 1 elements (e.g., > 1535 vs. 1536), an explicit IllegalArgumentException / ValidationException > is thrown immediately: > "VECTOR dimension mismatch: expected %d elements but got %d." > > *NULL Semantics* > > The Vector type itself is nullable, but. it can't contain NULL elements. > This means that represents dense embedding values. At the Flink SQL level, > it is represented as ARRAY<FLOAT NOT NULL>. > > *Backwards compatibility with Older clients* > > Well, right now it would throw an IllegalArgumentException. How is this > usually handled in Fluss? Via feature flags or is there another mechanism? > Also, now that 1.0 is about to be released maybe older clients can fail in > this case given that this would anyways be released in 1.0+ version? We > still need to solve this within the 1.0 version range though. Also, at the > flink-SQL layer, I don't expect any issues because those would still > represent the vector as ARRAY<FLOAT> (not sure about NOT NULL today). > > *Formal TYPEID Assignment* > > Thanks for the suggestion. Would call it out in the FIP and maybe send an > initial PR just adding this. > > *Nested Types and Matrices* > > ARRAY<VECTOR(n)> is syntactically valid and supported across the Fluss type > system, DataTypeParser, JSON SerDe, row formats (CompactedRow, IndexedRow), > and Arrow schema (List<FixedSizeList<Float32>(n)>) which means one can > store, serialize, and read nested vectors / matrices without issue. > However, Vector similarity search UDFs (e.g., cosine distance, L2 distance) > and lakehouse vector indexing (ANN/KNN) strictly target top-level 1D > VECTOR(n) columns. Matrix math, tensor operations, and multidimensional > vector indexing over nested ARRAY<VECTOR(n)> are left for future versions. > > *Lakehouse Integrations* > > For lakehouse formats that support a native vector type (e.g. Lance and > possibly Apache Paimon), Fluss VECTOR(n) will map directly to the engine's > native vector / fixed-size array type to support cold-tier vector indexing. > For lake formats without native vector support (e.g., Iceberg / standard > Parquet), VECTOR(n) will fall back to ARRAY<FLOAT NOT NULL>. > > *Open Question / Discussion Point:* > > How should hybrid/union reads behave when combining hot Fluss data > (VECTOR(n)) with cold tiers that only support a generic ARRAY<FLOAT>? > Should the reader runtime automatically wrap and validate dimensions on the > fly, or should we expose the schema as ARRAY<FLOAT> during hybrid scans to > keep the reader lightweight? > > Let me know if these make sense. Happy to address more questions! > > And once we have these things sorted, I can send a FIP across. > > Sagar. > > > > > > > > On Fri, Aug 28, 2026 at 12:44 PM Sagar <[email protected]> wrote: > > > Thanks Mehul for the feedback. I will address these questions; I think > > some of these aspects are covered in the POC but I need to document them. > > > > Should I wait for these questions to be resolved before starting the FIP > > or if the POC looks okay, can I write the FIP and address all these > > questions within it? > > > > Also, unfortunately, I won't be able to join the community call this time > > due to a conflict. > > > > Sagar. > > > > > > On Mon, Aug 24, 2026 at 11:23 PM Mehul Batra <[email protected]> > > wrote: > > > >> Thank you Sagar for putting this all together. I went through the > proposal > >> and the code, and I have a few questions from my side that I think are > >> important to clarify before we move forward: > >> > >> 1. > >> > >> *Zero-copy:* I think zero-copy is a good direction, especially for > the > >> Fluss → Arrow → Lance path. Could we clarify the scope of this claim? > >> Through Flink SQL, the data may still be materialized into Flink's > own > >> representation, so it would be good to distinguish where we can > >> actually > >> guarantee zero-copy. > >> 2. > >> > >> *Dimension as part of the type:* I prefer keeping the dimension in > the > >> type itself, e.g. VECTOR(1536), rather than as a table property. This > >> makes dimensionality an explicit schema guarantee and avoids > downstream > >> systems having to infer it from the data. > >> 3. > >> > >> *Dimension validation:* What happens when a VECTOR(1536) column > >> receives > >> a vector with 1535 or 1537 elements? We should ideally have a clear > >> validation error at the write boundary rather than a failure later > >> during > >> Arrow/Lance conversion. > >> 4. > >> > >> *Null semantics:* Could we clarify how nulls are handled? In > >> particular, > >> do we support a completely NULL vector, and are individual elements > >> within a vector allowed to be NULL? These are two different cases and > >> may have different implications for Arrow and Lance. I noticed the > POC > >> has > >> testNullVectorTiering, so it would be good to understand exactly what > >> case is covered. > >> 5. > >> > >> *Compatibility:* Since this introduces a new type ID, how do older > >> clients behave when they encounter a VECTOR column? We should make > sure > >> this is a clear and predictable compatibility failure rather than an > >> opaque > >> error. > >> 6. > >> > >> *Type ID:* The proposal currently mentions something like "e.g. 16". > I > >> think we should formally reserve the type ID before merging so there > >> is no > >> possibility of conflicts with other changes. > >> 7. > >> > >> *Nesting/matrices:* Since matrices are explicitly out of scope, can > we > >> confirm whether ARRAY<VECTOR(1536)> is actually supported in v1? > >> Otherwise, the suggested workaround of wrapping vectors in an array > >> would > >> not be valid. > >> 8. > >> > >> *Other lakehouse integrations:* It would be good to clarify how > VECTOR > >> is expected to behave across the different lakehouse paths, > >> particularly > >> where the hot tier has a fixed-size vector but the cold tier may only > >> support a list representation. We should make sure the type semantics > >> remain consistent across tiering and union reads. > >> > >> Also, I think this would be a great topic to bring to the community > call. > >> There are several cross-component implications here, and getting more > eyes > >> from the community would help us validate the design and catch any > >> compatibility or integration concerns early. > >> > >> Best Regards, > >> Mehul Batra > >> > > >
