[
https://issues.apache.org/jira/browse/SPARK-58787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uros Stankovic updated SPARK-58787:
-----------------------------------
Affects Version/s: 5.0.0
(was: 4.3.0)
> SPIP: Add a fixed-dimension numeric VECTOR data type
> ----------------------------------------------------
>
> Key: SPARK-58787
> URL: https://issues.apache.org/jira/browse/SPARK-58787
> Project: Spark
> Issue Type: Umbrella
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Uros Stankovic
> Priority: Major
> Labels: SPIP, releasenotes
>
> *Q1. What are you trying to do? Articulate your objectives using absolutely
> no jargon.*
> Add a new Spark SQL data type for storing a fixed number of numeric values.
> For example:
> {code:sql}
> CREATE TABLE documents (
> id BIGINT,
> embedding VECTOR(768, FLOAT)
> );
> {code}
> {{VECTOR(768, FLOAT)}} means that every non-null value contains exactly 768
> non-null FLOAT coordinates. The vector column itself may be nullable, but a
> present vector value with the wrong number of coordinates or a null
> coordinate is invalid.
> The initial supported coordinate types will be:
> * INT
> * BIGINT
> * FLOAT
> * DOUBLE
> The dimension must be a positive integer and is part of the data type. For
> example, {{VECTOR(768, FLOAT)}} and {{VECTOR(1536, FLOAT)}} are different
> types.
> The new type represents dense vectors. It is intended for embeddings,
> numerical feature vectors, count vectors, and interoperability with systems
> and storage formats that have a fixed-dimension vector type.
> The SQL syntax follows the emerging SQL standard proposal, which places the
> dimension before the coordinate type:
> {code:sql}
> VECTOR(<dimension>, <coordinate type>)
> {code}
> The corresponding programmatic Spark type will be {{VectorType(FloatType,
> 768)}}.
> The type will use Spark's existing array representation during execution.
> This proposal introduces a new logical type and schema contract, not a new
> packed in-memory representation.
> *Q2. What problem is this proposal NOT designed to solve?*
> This proposal does not attempt to:
> * Add a generic fixed-size array for arbitrary element types.
> * Represent matrices or multidimensional tensors.
> * Represent sparse vectors.
> * Replace the existing MLlib {{VectorUDT}}.
> * Add a vector index or approximate-nearest-neighbor search engine.
> * Add a complete library of vector distance and similarity functions.
> * Introduce a specialized UnsafeRow or columnar representation.
> * Add FP16, BF16, INT8, INT4, or binary coordinate types in the first version.
> * Make quantization performed internally by an index part of the logical
> column type.
> * Require coordinates to be finite, normalized, or to have a non-zero norm as
> part of the base type.
> Finiteness and normalization are data-quality or operation-specific
> constraints. For example, an index or cosine-distance function may reject
> NaN, infinity, or a zero vector even though those values can be represented
> by {{VECTOR(N, FLOAT)}}.
> Low-precision types may be proposed later. FP16, BF16, and INT8 require
> additional primitive-type, casting, client, and storage-format work. INT8
> quantization may also require scale and zero-point metadata that cannot be
> represented by the coordinate type alone.
> *Q3. How is it done today, and what are the limits of current practice?*
> Dense vectors are normally stored as numeric arrays:
> {code:sql}
> CREATE TABLE documents (
> embedding ARRAY<FLOAT>
> );
> {code}
> Spark's {{ArrayType}} can specify whether elements are nullable, but it
> cannot declare a fixed dimension or distinguish a vector from an ordinary
> list. Different-length values can therefore be stored in the same column, and
> the dimension must be configured again in each downstream consumer.
> Users can approximate the contract with checks or custom metadata, but those
> mechanisms do not establish a portable vector type. File formats, catalogs,
> clients, functions, and other engines still see an ordinary array. The
> dimension and vector identity can be lost across a system boundary, and
> malformed vectors are often detected by a downstream function or index
> builder instead of at construction or ingestion time.
> Spark also has MLlib {{VectorUDT}}, but it represents a different abstraction:
> ||SQL VECTOR||MLlib VectorUDT||
> |Dimension is part of the schema|Dimension belongs to each value|
> |Dense only|Dense or sparse|
> |INT, BIGINT, FLOAT, or DOUBLE|DOUBLE|
> |Native SQL data type|User-defined type|
> |Intended for SQL and storage interoperability|Intended for MLlib algorithms|
> |Backed by ArrayData|Encoded as a UDT struct|
> Reusing {{VectorUDT}} would not provide the required SQL or storage-format
> semantics.
> *Q4. What is new in your approach and why do you think it will be successful?*
> The proposal adds a native parameterized Spark SQL type:
> {code:scala}
> VectorType(elementType: DataType, dimension: Int)
> {code}
> The following invariants apply:
> # The dimension is positive.
> # Every non-null value has exactly the declared number of coordinates.
> # Coordinates cannot be null.
> # The coordinate type is INT, BIGINT, FLOAT, or DOUBLE.
> # Top-level nullability continues to be controlled by
> {{StructField.nullable}}.
> The type reuses the same internal representation as an equivalent array:
> {code}
> VECTOR(N, T)
> logical type: VectorType(T, N)
> physical value: ArrayData
> coordinates: existing Spark primitive representation
> {code}
> This avoids new execution kernels for basic projection, shuffle,
> serialization, caching, and columnar processing.
> h4. Casting
> Converting an array to a vector is explicit and validated:
> {code:sql}
> SELECT CAST(embedding AS VECTOR(768, FLOAT))
> FROM documents;
> {code}
> The cast rejects a wrong dimension, a null coordinate, or a coordinate that
> cannot be cast to the target numeric type. Errors should identify the column
> and expected and actual dimensions.
> Converting a vector to the corresponding array is lossless. Spark may apply
> that conversion implicitly when an existing array operation consumes a
> vector. An operation that cannot statically preserve the dimension returns
> ARRAY, not VECTOR.
> No implicit promotion between vector coordinate types is proposed initially.
> A user may request a value-level conversion explicitly, for example {{CAST(v
> AS VECTOR(768, DOUBLE))}}. In-place schema evolution is treated separately
> and is not allowed initially.
> h4. Interoperability
> The initial coordinate types map directly to the proposed Apache Iceberg
> vector types:
> ||Spark||Iceberg||
> |INT|int|
> |BIGINT|long|
> |FLOAT|float|
> |DOUBLE|double|
> The proposal also aligns with Arrow {{FixedSizeList}} and active Parquet
> fixed-size-list/vector design work.
> Spark will not invent a private Parquet representation. The canonical mapping
> will follow the representation accepted by the Parquet community. Ordinary
> Parquet LIST columns will not be inferred to be vectors merely because
> observed values have the same length.
> For a format that cannot preserve the vector type, Spark must reject the
> operation clearly or require an explicit cast to ARRAY. It must not claim to
> preserve VECTOR while silently discarding its dimension.
> This approach should be successful because it builds on existing Spark array
> execution paths, moves malformed-value detection closer to ingestion, aligns
> with SQL and open-format work, and leaves physical optimization and vector
> indexing independent from the logical type.
> *Q5. Who cares? If you are successful, what difference will it make?*
> h4. Embedding and vector-search users
> Embeddings are commonly represented as fixed-length arrays of 32-bit
> floating-point values. {{VECTOR(N, FLOAT)}} records that contract directly in
> the schema. Writers can reject malformed values immediately, and future
> functions or indexes can discover the dimension from the schema instead of
> requiring it again.
> h4. Spark ML and numerical users
> DOUBLE is useful for interoperability with numerical and classical ML
> systems. Spark MLlib vectors are Double-valued, and systems such as BigQuery
> represent vector-search inputs as {{ARRAY<FLOAT64>}}. {{VECTOR(N, DOUBLE)}}
> preserves those values without an obligatory narrowing conversion.
> h4. Integer-vector users
> INT and BIGINT support count vectors, fixed histograms, integer-valued
> features, and lossless interchange with schemas that use 32-bit or 64-bit
> coordinates. BIGINT is not expected to be a common embedding type; its
> primary justification is lossless interoperability and alignment with
> numeric-vector type systems.
> h4. Data-source and library developers
> A native type gives data sources, catalogs, functions, and libraries an
> unambiguous signal that a column is a dense fixed-dimension numeric vector.
> Projects can add vector operations without guessing from column names,
> observed array lengths, or custom metadata.
> *Q6. What are the risks?*
> h4. Broad type-system integration
> A native type must be handled across Catalyst, encoders, code generation,
> columnar execution, schema serialization, data sources, Spark Connect,
> PySpark, Arrow, JDBC, and other clients. Missing handling may cause incorrect
> fallback behavior.
> Mitigation: split implementation into milestones and add type-coverage tests
> similar to other native parameterized types.
> h4. Confusion with MLlib vectors
> Spark already exposes {{org.apache.spark.ml.linalg.Vector}}, {{VectorUDT}},
> and {{SQLDataTypes.VectorType}}. The proposed class is
> {{org.apache.spark.sql.types.VectorType}} and has different semantics.
> Documentation must distinguish them clearly. No implicit conversion with
> MLlib is proposed initially.
> h4. Validation cost
> An ARRAY-to-VECTOR cast must inspect each value to validate its dimension and
> null coordinates. Readers can avoid repeated validation when a trusted
> physical format provides a fixed-size representation with required
> coordinates.
> h4. Evolving storage specifications
> The Parquet and Iceberg proposals may change during implementation.
> Mitigation: keep the Spark logical contract independent of a physical
> encoding and avoid a Spark-private encoding.
> h4. Old-reader compatibility
> Older Spark versions and protocol clients will not recognize VectorType. The
> implementation must define explicit compatibility behavior and must not
> silently reinterpret an unsupported logical type.
> h4. Scope expansion
> Vector types are often discussed together with distance functions, ANN
> indexes, tensors, and quantization. The initial work will remain limited to
> the type system, validation, core interoperability, and persistence.
> *Q7. How long will it take?*
> The expected implementation time is approximately six to nine months, with
> work proceeding in parallel and persistence depending on the corresponding
> file-format specifications.
> # Base type functionality - 4 to 6 weeks
> ## Add VectorType, SQL parser and DDL support.
> ## Add JSON and catalog serialization.
> ## Add Scala, Java, and PySpark type APIs.
> # Casts and Catalyst integration - 4 to 6 weeks
> ## Add validated ARRAY-to-VECTOR casts and lossless VECTOR-to-ARRAY
> conversion.
> ## Define common-type and expression behavior.
> ## Integrate interpreted and code-generated execution.
> ## Add structured errors.
> # Persistence and Arrow integration - 2 to 3 months
> ## Add the standardized Parquet mapping.
> ## Add Arrow FixedSizeList conversion.
> ## Add columnar reader/writer and data-source conversion support.
> ## Define behavior for unsupported formats.
> # Client support - 1 to 2 months
> ## Spark Connect protocol and clients.
> ## PySpark, JDBC, and Hive/Thrift metadata.
> ## Pandas and Arrow UDF paths where applicable.
> # Testing, documentation, compatibility, and benchmarks - 1 month
> Vector functions, indexes, MLlib conversions, and low-precision coordinate
> types will be follow-up issues.
> *Q8. What are the mid-term and final "exams" to check for success?*
> h4. Mid-term criteria
> * Users can declare {{VECTOR(N, T)}} in SQL and construct the equivalent type
> in Scala, Java, and PySpark.
> * The type survives schema JSON and catalog serialization.
> * ARRAY-to-VECTOR casts reject wrong dimensions and null coordinates with
> structured errors.
> * Execution reuses ArrayData.
> * Projection, filtering, shuffle, caching, and columnar transitions preserve
> the type and values.
> * The distinction from MLlib VectorUDT is documented and tested.
> h4. Final criteria
> * Spark can read and write the standardized Parquet representation.
> * Arrow conversion uses FixedSizeList.
> * Spark Connect, PySpark, and JDBC preserve or report the coordinate type and
> dimension correctly.
> * Spark maps all four initial coordinate types to the corresponding Iceberg
> vector types.
> * Unsupported formats fail clearly or require an explicit cast to ARRAY.
> * Existing ARRAY behavior and performance do not regress.
> * Vector scans and projections perform comparably to equivalent numeric
> arrays.
> * Cross-engine tests preserve element type, dimension, top-level nullability,
> and non-null coordinate semantics.
> *Appendix A. Proposed API Changes.*
> h4. SQL type syntax
> {code:sql}
> VECTOR(<dimension>, <coordinate type>)
> {code}
> Examples:
> {code:sql}
> VECTOR(3, INT)
> VECTOR(128, BIGINT)
> VECTOR(768, FLOAT)
> VECTOR(1536, DOUBLE)
> {code}
> No default coordinate type is proposed. Both parameters are required.
> h4. Scala API
> Conceptually:
> {code:scala}
> package org.apache.spark.sql.types
> case class VectorType(
> elementType: DataType,
> dimension: Int) extends DataType
> {code}
> Example:
> {code:scala}
> val schema = StructType(Seq(
> StructField("embedding", VectorType(FloatType, 768), nullable = true)
> ))
> {code}
> h4. Java API
> {code:java}
> DataType vectorType =
> DataTypes.createVectorType(DataTypes.FloatType, 768);
> {code}
> h4. PySpark API
> {code:python}
> from pyspark.sql.types import FloatType, StructField, StructType, VectorType
> schema = StructType([
> StructField("embedding", VectorType(FloatType(), 768), nullable=True)
> ])
> {code}
> Values use the same external representation as the corresponding numeric
> array. No new user-facing vector value class is required initially.
> h4. Type serialization
> Proposed JSON:
> {code:json}
> {
> "type": "vector",
> "elementType": "float",
> "dimension": 768
> }
> {code}
> Proposed canonical type string: {{vector(768,float)}}.
> *Appendix B. Type System.*
> h4. Type identity
> Both coordinate type and dimension are part of type identity:
> {code}
> VECTOR(768, FLOAT) != VECTOR(1536, FLOAT)
> VECTOR(768, FLOAT) != VECTOR(768, DOUBLE)
> {code}
> h4. Coordinate types
> ||SQL spelling||Spark type||
> |INT or INTEGER|IntegerType|
> |BIGINT|LongType|
> |FLOAT or REAL|FloatType|
> |DOUBLE or DOUBLE PRECISION|DoubleType|
> DECIMAL, strings, booleans, intervals, and nested types are not supported
> initially.
> h4. Nullability
> VectorType has no {{containsNull}} parameter. Coordinate nullability is
> always false. Top-level nullability remains a property of StructField.
> h4. Floating-point special values
> NaN and positive or negative infinity remain representable, following
> FloatType and DoubleType. Individual functions, writers, indexes, or table
> constraints may impose a finite-value requirement.
> h4. Casting
> ||Source||Target||Behavior||
> |ARRAY<T>|VECTOR(N,T)|Explicit; validates dimension and null coordinates|
> |ARRAY<S>|VECTOR(N,T)|Explicit; applies normal numeric casting and vector
> validation|
> |VECTOR(N,T)|ARRAY<T>|Lossless|
> |VECTOR(N,S)|VECTOR(N,T)|Explicit numeric cast|
> |VECTOR(N,T)|VECTOR(M,T), N != M|Rejected at analysis time|
> No implicit vector-to-vector coordinate promotion is proposed initially.
> h4. Comparison
> Equality and hashing use element-by-element semantics consistent with the
> equivalent Spark array. Ordering comparisons and ORDER BY on a vector are
> unsupported initially. Vector columns are not supported as table partition
> columns, bucket columns, map keys, or range-clustering keys initially.
> h4. Schema evolution
> Changing the dimension is incompatible. Changing the coordinate type is also
> incompatible as an in-place schema evolution operation initially, even where
> an explicit value-level cast exists. Top-level nullability follows existing
> Spark field-nullability rules.
> *Appendix C. Internal and Storage Representations.*
> h4. In-memory representation
> A vector value uses ArrayData and the existing primitive coordinate
> representation. Optimizer, UnsafeRow, shuffle, cache, and columnar execution
> paths can reuse existing array machinery while retaining VectorType in the
> logical schema.
> h4. Arrow
> The canonical Arrow mapping is {{VectorType(T, N) <-> FixedSizeList<T, N>}}
> with a non-nullable child field.
> h4. Parquet
> The canonical mapping will follow the accepted Parquet fixed-size-list/vector
> representation. It must preserve dimension, coordinate type, and required
> coordinates. An ordinary LIST is not inferred to be a vector; legacy data
> requires an explicit validated cast. Spark will not introduce a private
> Parquet annotation as part of this SPIP.
> h4. Iceberg
> Expected mapping:
> {code}
> VectorType(IntegerType, N) -> int[N]
> VectorType(LongType, N) -> long[N]
> VectorType(FloatType, N) -> float[N]
> VectorType(DoubleType, N) -> double[N]
> {code}
> The final serialized spelling is controlled by the Iceberg specification.
> h4. Spark Connect
> Conceptually, Spark Connect will add:
> {code:protobuf}
> message Vector {
> DataType element_type = 1;
> int32 dimension = 2;
> }
> {code}
> *Appendix D. Relationship to MLlib.*
> The new type does not replace or change VectorUDT. A future follow-up may add
> explicit conversion between a dense MLlib Double vector and {{VECTOR(N,
> DOUBLE)}}. Conversion from a sparse MLlib vector requires explicit
> densification and is outside this proposal.
> *Appendix E. Alternatives Considered.*
> h4. Continue using ARRAY
> Rejected because an array does not declare a dimension or vector identity.
> Metadata and checks are not portable enough for catalogs, clients, file
> formats, and function resolution.
> h4. Add a FLOAT-only vector
> Rejected because it prevents lossless interoperability with Double-valued
> numerical systems and does not align with the initial Iceberg numeric-vector
> proposal. FLOAT remains the primary expected embedding type.
> h4. Reuse MLlib VectorUDT
> Rejected because it is Double-only, supports sparse and variable-dimension
> values, has a UDT struct encoding, and is not a native SQL or open-format
> vector type.
> h4. Add a generic FIXED_SIZE_ARRAY<T,N>
> Rejected for the initial proposal because it introduces a broader type
> without a demonstrated non-numeric use case. A generic fixed-size container
> could be proposed independently later.
> h4. Add a specialized packed physical representation
> Rejected initially because it would require new row, columnar, shuffle,
> code-generation, and storage paths. Reusing ArrayData provides the logical
> benefits with substantially less risk.
> h4. Include FP16, BF16, and INT8 initially
> Deferred because Spark and Iceberg do not currently expose all of them as
> ordinary primitive types. FP16 is a strong future coordinate type; BF16 is
> useful for model interoperability; INT8 is useful for quantized search but
> may require additional quantization metadata.
> *References*
> * [SQL:202y vector proposal
> overview|https://peter.eisentraut.org/blog/2025/06/24/waiting-for-sql-202y-vectors]
> * [Iceberg Vector Type Support
> Proposal|https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit]
> * [Parquet fixed-size vector/list
> proposal|https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit]
> * [Apache Arrow Fixed-Size List
> layout|https://arrow.apache.org/docs/format/Columnar.html#fixed-size-list-layout]
> * [Spark MLlib Vector
> API|https://spark.apache.org/docs/latest/api/java/org/apache/spark/ml/linalg/Vector.html]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
