Thanks Mehul :) I will try to join the community call this month. Sagar.
On Mon, 7 Sep 2026 at 1:18 PM, Mehul Batra <[email protected]> wrote: > Hi Sagar, thanks for the detailed responses. I’ve gone through the email > and will take some time to review everything properly. I’ll get back to you > once I’ve had a chance to digest the details. > > I’d also suggest joining the community call so we can discuss this with the > broader community. You could walk us through the design decisions you’ve > made in the repo and capture them in a Google Doc. Once we’ve discussed and > aligned on those points, we can move forward with the FIP. > > Best Regards, > Mehul Batra > > On Sun, Aug 30, 2026 at 9:30 PM Sagar <[email protected]> wrote: > > > Hi Mehul, > > > > PFA my responses to the questions: > > > > *Zero-Copy Scope:* > > > > Zero-copy is guaranteed for the native columnar tiering pipeline (Fluss > > Arrow Log-> Arrow FixedSizeListVector-> Lance Dataset). Row-oriented > > writes and Flink SQL query execution follow standard row/array > > serialization (InternalArray<-> Flink ArrayData). > > > > *Dimension as part of Datatype:* > > > > The POC already implements dimension as a first class parameter of > > VectorType, i.e you specify it with Vector(n). > > > > *Dimension Validation at Write boundary:* > > > > Validation is enforced at the client write boundary (AppendWriter, > > UpsertWriter, BinaryWriter, and Flink Sink serializer) as well as the > Arrow > > batch writer (ArrowVectorWriter). When writing to a VECTOR(n) column, the > > writer verifies array.size() == n before serializing or sending data. > > > > Failure mode: If an incoming vector has n − 1 or n + 1 elements (e.g., > > 1535 vs. 1536), an explicit IllegalArgumentException / > ValidationException > > is thrown immediately: > > "VECTOR dimension mismatch: expected %d elements but got %d." > > > > *NULL Semantics* > > > > The Vector type itself is nullable, but. it can't contain NULL elements. > > This means that represents dense embedding values. At the Flink SQL > level, > > it is represented as ARRAY<FLOAT NOT NULL>. > > > > *Backwards compatibility with Older clients* > > > > Well, right now it would throw an IllegalArgumentException. How is this > > usually handled in Fluss? Via feature flags or is there another > mechanism? > > Also, now that 1.0 is about to be released maybe older clients can fail > in > > this case given that this would anyways be released in 1.0+ version? We > > still need to solve this within the 1.0 version range though. Also, at > the > > flink-SQL layer, I don't expect any issues because those would still > > represent the vector as ARRAY<FLOAT> (not sure about NOT NULL today). > > > > *Formal TYPEID Assignment* > > > > Thanks for the suggestion. Would call it out in the FIP and maybe send an > > initial PR just adding this. > > > > *Nested Types and Matrices* > > > > ARRAY<VECTOR(n)> is syntactically valid and supported across the Fluss > type > > system, DataTypeParser, JSON SerDe, row formats (CompactedRow, > IndexedRow), > > and Arrow schema (List<FixedSizeList<Float32>(n)>) which means one can > > store, serialize, and read nested vectors / matrices without issue. > > However, Vector similarity search UDFs (e.g., cosine distance, L2 > distance) > > and lakehouse vector indexing (ANN/KNN) strictly target top-level 1D > > VECTOR(n) columns. Matrix math, tensor operations, and multidimensional > > vector indexing over nested ARRAY<VECTOR(n)> are left for future > versions. > > > > *Lakehouse Integrations* > > > > For lakehouse formats that support a native vector type (e.g. Lance and > > possibly Apache Paimon), Fluss VECTOR(n) will map directly to the > engine's > > native vector / fixed-size array type to support cold-tier vector > indexing. > > For lake formats without native vector support (e.g., Iceberg / standard > > Parquet), VECTOR(n) will fall back to ARRAY<FLOAT NOT NULL>. > > > > *Open Question / Discussion Point:* > > > > How should hybrid/union reads behave when combining hot Fluss data > > (VECTOR(n)) with cold tiers that only support a generic ARRAY<FLOAT>? > > Should the reader runtime automatically wrap and validate dimensions on > the > > fly, or should we expose the schema as ARRAY<FLOAT> during hybrid scans > to > > keep the reader lightweight? > > > > Let me know if these make sense. Happy to address more questions! > > > > And once we have these things sorted, I can send a FIP across. > > > > Sagar. > > > > > > > > > > > > > > > > On Fri, Aug 28, 2026 at 12:44 PM Sagar <[email protected]> > wrote: > > > > > Thanks Mehul for the feedback. I will address these questions; I think > > > some of these aspects are covered in the POC but I need to document > them. > > > > > > Should I wait for these questions to be resolved before starting the > FIP > > > or if the POC looks okay, can I write the FIP and address all these > > > questions within it? > > > > > > Also, unfortunately, I won't be able to join the community call this > time > > > due to a conflict. > > > > > > Sagar. > > > > > > > > > On Mon, Aug 24, 2026 at 11:23 PM Mehul Batra <[email protected] > > > > > wrote: > > > > > >> Thank you Sagar for putting this all together. I went through the > > proposal > > >> and the code, and I have a few questions from my side that I think are > > >> important to clarify before we move forward: > > >> > > >> 1. > > >> > > >> *Zero-copy:* I think zero-copy is a good direction, especially for > > the > > >> Fluss → Arrow → Lance path. Could we clarify the scope of this > claim? > > >> Through Flink SQL, the data may still be materialized into Flink's > > own > > >> representation, so it would be good to distinguish where we can > > >> actually > > >> guarantee zero-copy. > > >> 2. > > >> > > >> *Dimension as part of the type:* I prefer keeping the dimension in > > the > > >> type itself, e.g. VECTOR(1536), rather than as a table property. > This > > >> makes dimensionality an explicit schema guarantee and avoids > > downstream > > >> systems having to infer it from the data. > > >> 3. > > >> > > >> *Dimension validation:* What happens when a VECTOR(1536) column > > >> receives > > >> a vector with 1535 or 1537 elements? We should ideally have a clear > > >> validation error at the write boundary rather than a failure later > > >> during > > >> Arrow/Lance conversion. > > >> 4. > > >> > > >> *Null semantics:* Could we clarify how nulls are handled? In > > >> particular, > > >> do we support a completely NULL vector, and are individual elements > > >> within a vector allowed to be NULL? These are two different cases > and > > >> may have different implications for Arrow and Lance. I noticed the > > POC > > >> has > > >> testNullVectorTiering, so it would be good to understand exactly > what > > >> case is covered. > > >> 5. > > >> > > >> *Compatibility:* Since this introduces a new type ID, how do older > > >> clients behave when they encounter a VECTOR column? We should make > > sure > > >> this is a clear and predictable compatibility failure rather than > an > > >> opaque > > >> error. > > >> 6. > > >> > > >> *Type ID:* The proposal currently mentions something like "e.g. > 16". > > I > > >> think we should formally reserve the type ID before merging so > there > > >> is no > > >> possibility of conflicts with other changes. > > >> 7. > > >> > > >> *Nesting/matrices:* Since matrices are explicitly out of scope, can > > we > > >> confirm whether ARRAY<VECTOR(1536)> is actually supported in v1? > > >> Otherwise, the suggested workaround of wrapping vectors in an array > > >> would > > >> not be valid. > > >> 8. > > >> > > >> *Other lakehouse integrations:* It would be good to clarify how > > VECTOR > > >> is expected to behave across the different lakehouse paths, > > >> particularly > > >> where the hot tier has a fixed-size vector but the cold tier may > only > > >> support a list representation. We should make sure the type > semantics > > >> remain consistent across tiering and union reads. > > >> > > >> Also, I think this would be a great topic to bring to the community > > call. > > >> There are several cross-component implications here, and getting more > > eyes > > >> from the community would help us validate the design and catch any > > >> compatibility or integration concerns early. > > >> > > >> Best Regards, > > >> Mehul Batra > > >> > > > > > >
