Hi Sagar, I have gone through your responses properly now. Most of it is settled from my side: zero-copy scoped to the columnar tiering path, dimension as a first-class type parameter, the type ID PR, and the Iceberg fallback to ARRAY<FLOAT NOT NULL> looks alright.
On scope, before you start writing. Let's keep this first FIP to the vector data type only. Similarity UDFs, ANN and KNN indexing, hybrid read semantics and multi-dimensional support each come after, as their own proposals. The type is close to done and should pass quickly, and I would rather not have it wait behind designs that will take several rounds. So the FIP covers: VECTOR(n) with fixed dimension and element type at schema time, the JSON descriptor and type root constant, null semantics, dimension validation at the write boundary, the row format encodings, Flink and Spark mappings, ARRAY<VECTOR(n)> in the type system, schema evolution rules, and client compatibility. Four things I would still tighten: 1. Validation sites. AppendWriter, UpsertWriter, BinaryWriter, the Flink sink serializer and ArrowVectorWriter is five chances to miss one and five messages that will diverge. One canonical check with the rest delegating to it, plus a test that writes through the raw Java client rather than Flink, since that is the path most likely to be missed. Related: does the server validate, or trust the client? On the Arrow path the length is structural so a bad write cannot happen, but CompactedRow and IndexedRow carry values without that guarantee. 2. ARRAY<FLOAT NOT NULL> at the Flink layer. You flagged you were unsure and I think this bites. Most pipelines produce nullable elements, and Flink does not implicitly coerce those to non-nullable, so inserts may fail at plan time until the user adds a cast. Worth running INSERT INTO t SELECT ARRAY[1.0, 2.0] before the FIP claims the path works. If it fails, the FIP should say whether we add coercion or document the cast. Separately, a null element should throw the same explicit error as a dimension mismatch, not an NPE in the converter. 3. Type ID. The doc says 15 (BYTES) is the last number, but ARRAY, MAP and ROW exist too, so they must have numbers as well. Either the list already goes past 15, or nested types work some other way. Worth checking the code and writing down the exact number you are taking. Not "next available", because if another change lands first, two branches end up claiming the same number. 4. Nested vectors and tiering. ARRAY<VECTOR(n)> works in the type system, but there is no multi-level Lance conversion behind it yet. So a table with that column and table.datalake.enabled set should fail at CREATE TABLE rather than accept writes and surface a background error from the tiering service later. Rejecting is cleaner than leaving it undefined. On backwards compatibility, I would not invent a new mechanism. ARRAY, MAP and ROW were each added as new type roots at some point, so there is already a precedent in the codebase for what an old client does with a type it cannot parse. Worth following that rather than adding feature flags, which do not help here since the table either has the column or it does not. My expectation is we keep the failure but make it a readable error naming the table, the type and the minimum client version. One thing to state in the FIP: which release this ships in. That decides how much compatibility work there is. If it lands in 1.0, the only thing that breaks is an old 0.x client against a 1.0 server, and breaking that at a major version is normal. If it lands in 1.1, then a 1.0 client breaks against a 1.1 table, and that is inside the same major version, which is a much bigger ask. On your union read question, so you are not blocked. A table should look the same no matter where its data currently sits. If it shows VECTOR(n) for fresh data and ARRAY<FLOAT> once that data moves to the lake, the same table has two different schemas depending on the query, and everything downstream has to handle both. So I would show VECTOR(n) always and convert the cold data back into a vector on read. On checking every row, I would not. We already checked the size when we wrote it, so checking again on read costs time for something we already know. The one exception is if something other than Fluss can write into that lake table, because then we cannot trust it. Either way, that is a question for the later read proposal, not this FIP. Overall this is in good shape. Go ahead and start the FIP, focused on the vector data type. Present it on the community call and open the discussion thread from there. Best Regards, Mehul Batra
