Uros Stankovic created SPARK-58787:
--------------------------------------
Summary: SPIP: Add a fixed-dimension numeric VECTOR data type
Key: SPARK-58787
URL: https://issues.apache.org/jira/browse/SPARK-58787
Project: Spark
Issue Type: Umbrella
Components: SQL
Affects Versions: 4.3.0
Reporter: Uros Stankovic
*Q1. What are you trying to do? Articulate your objectives using absolutely no
jargon.*
Add a new Spark SQL data type for storing a fixed number of numeric values.
For example:
{code:sql}
CREATE TABLE documents (
id BIGINT,
embedding VECTOR(768, FLOAT)
);
{code}
{{VECTOR(768, FLOAT)}} means that every non-null value contains exactly 768
non-null FLOAT coordinates. The vector column itself may be nullable, but a
present vector value with the wrong number of coordinates or a null coordinate
is invalid.
The initial supported coordinate types will be:
* INT
* BIGINT
* FLOAT
* DOUBLE
The dimension must be a positive integer and is part of the data type. For
example, {{VECTOR(768, FLOAT)}} and {{VECTOR(1536, FLOAT)}} are different types.
The new type represents dense vectors. It is intended for embeddings, numerical
feature vectors, count vectors, and interoperability with systems and storage
formats that have a fixed-dimension vector type.
The SQL syntax follows the emerging SQL standard proposal, which places the
dimension before the coordinate type:
{code:sql}
VECTOR(<dimension>, <coordinate type>)
{code}
The corresponding programmatic Spark type will be {{VectorType(FloatType,
768)}}.
The type will use Spark's existing array representation during execution. This
proposal introduces a new logical type and schema contract, not a new packed
in-memory representation.
*Q2. What problem is this proposal NOT designed to solve?*
This proposal does not attempt to:
* Add a generic fixed-size array for arbitrary element types.
* Represent matrices or multidimensional tensors.
* Represent sparse vectors.
* Replace the existing MLlib {{VectorUDT}}.
* Add a vector index or approximate-nearest-neighbor search engine.
* Add a complete library of vector distance and similarity functions.
* Introduce a specialized UnsafeRow or columnar representation.
* Add FP16, BF16, INT8, INT4, or binary coordinate types in the first version.
* Make quantization performed internally by an index part of the logical column
type.
* Require coordinates to be finite, normalized, or to have a non-zero norm as
part of the base type.
Finiteness and normalization are data-quality or operation-specific
constraints. For example, an index or cosine-distance function may reject NaN,
infinity, or a zero vector even though those values can be represented by
{{VECTOR(N, FLOAT)}}.
Low-precision types may be proposed later. FP16, BF16, and INT8 require
additional primitive-type, casting, client, and storage-format work. INT8
quantization may also require scale and zero-point metadata that cannot be
represented by the coordinate type alone.
*Q3. How is it done today, and what are the limits of current practice?*
Dense vectors are normally stored as numeric arrays:
{code:sql}
CREATE TABLE documents (
embedding ARRAY<FLOAT>
);
{code}
Spark's {{ArrayType}} can specify whether elements are nullable, but it cannot
declare a fixed dimension or distinguish a vector from an ordinary list.
Different-length values can therefore be stored in the same column, and the
dimension must be configured again in each downstream consumer.
Users can approximate the contract with checks or custom metadata, but those
mechanisms do not establish a portable vector type. File formats, catalogs,
clients, functions, and other engines still see an ordinary array. The
dimension and vector identity can be lost across a system boundary, and
malformed vectors are often detected by a downstream function or index builder
instead of at construction or ingestion time.
Spark also has MLlib {{VectorUDT}}, but it represents a different abstraction:
||SQL VECTOR||MLlib VectorUDT||
|Dimension is part of the schema|Dimension belongs to each value|
|Dense only|Dense or sparse|
|INT, BIGINT, FLOAT, or DOUBLE|DOUBLE|
|Native SQL data type|User-defined type|
|Intended for SQL and storage interoperability|Intended for MLlib algorithms|
|Backed by ArrayData|Encoded as a UDT struct|
Reusing {{VectorUDT}} would not provide the required SQL or storage-format
semantics.
*Q4. What is new in your approach and why do you think it will be successful?*
The proposal adds a native parameterized Spark SQL type:
{code:scala}
VectorType(elementType: DataType, dimension: Int)
{code}
The following invariants apply:
# The dimension is positive.
# Every non-null value has exactly the declared number of coordinates.
# Coordinates cannot be null.
# The coordinate type is INT, BIGINT, FLOAT, or DOUBLE.
# Top-level nullability continues to be controlled by {{StructField.nullable}}.
The type reuses the same internal representation as an equivalent array:
{code}
VECTOR(N, T)
logical type: VectorType(T, N)
physical value: ArrayData
coordinates: existing Spark primitive representation
{code}
This avoids new execution kernels for basic projection, shuffle, serialization,
caching, and columnar processing.
h4. Casting
Converting an array to a vector is explicit and validated:
{code:sql}
SELECT CAST(embedding AS VECTOR(768, FLOAT))
FROM documents;
{code}
The cast rejects a wrong dimension, a null coordinate, or a coordinate that
cannot be cast to the target numeric type. Errors should identify the column
and expected and actual dimensions.
Converting a vector to the corresponding array is lossless. Spark may apply
that conversion implicitly when an existing array operation consumes a vector.
An operation that cannot statically preserve the dimension returns ARRAY, not
VECTOR.
No implicit promotion between vector coordinate types is proposed initially. A
user may request a value-level conversion explicitly, for example {{CAST(v AS
VECTOR(768, DOUBLE))}}. In-place schema evolution is treated separately and is
not allowed initially.
h4. Interoperability
The initial coordinate types map directly to the proposed Apache Iceberg vector
types:
||Spark||Iceberg||
|INT|int|
|BIGINT|long|
|FLOAT|float|
|DOUBLE|double|
The proposal also aligns with Arrow {{FixedSizeList}} and active Parquet
fixed-size-list/vector design work.
Spark will not invent a private Parquet representation. The canonical mapping
will follow the representation accepted by the Parquet community. Ordinary
Parquet LIST columns will not be inferred to be vectors merely because observed
values have the same length.
For a format that cannot preserve the vector type, Spark must reject the
operation clearly or require an explicit cast to ARRAY. It must not claim to
preserve VECTOR while silently discarding its dimension.
This approach should be successful because it builds on existing Spark array
execution paths, moves malformed-value detection closer to ingestion, aligns
with SQL and open-format work, and leaves physical optimization and vector
indexing independent from the logical type.
*Q5. Who cares? If you are successful, what difference will it make?*
h4. Embedding and vector-search users
Embeddings are commonly represented as fixed-length arrays of 32-bit
floating-point values. {{VECTOR(N, FLOAT)}} records that contract directly in
the schema. Writers can reject malformed values immediately, and future
functions or indexes can discover the dimension from the schema instead of
requiring it again.
h4. Spark ML and numerical users
DOUBLE is useful for interoperability with numerical and classical ML systems.
Spark MLlib vectors are Double-valued, and systems such as BigQuery represent
vector-search inputs as {{ARRAY<FLOAT64>}}. {{VECTOR(N, DOUBLE)}} preserves
those values without an obligatory narrowing conversion.
h4. Integer-vector users
INT and BIGINT support count vectors, fixed histograms, integer-valued
features, and lossless interchange with schemas that use 32-bit or 64-bit
coordinates. BIGINT is not expected to be a common embedding type; its primary
justification is lossless interoperability and alignment with numeric-vector
type systems.
h4. Data-source and library developers
A native type gives data sources, catalogs, functions, and libraries an
unambiguous signal that a column is a dense fixed-dimension numeric vector.
Projects can add vector operations without guessing from column names, observed
array lengths, or custom metadata.
*Q6. What are the risks?*
h4. Broad type-system integration
A native type must be handled across Catalyst, encoders, code generation,
columnar execution, schema serialization, data sources, Spark Connect, PySpark,
Arrow, JDBC, and other clients. Missing handling may cause incorrect fallback
behavior.
Mitigation: split implementation into milestones and add type-coverage tests
similar to other native parameterized types.
h4. Confusion with MLlib vectors
Spark already exposes {{org.apache.spark.ml.linalg.Vector}}, {{VectorUDT}}, and
{{SQLDataTypes.VectorType}}. The proposed class is
{{org.apache.spark.sql.types.VectorType}} and has different semantics.
Documentation must distinguish them clearly. No implicit conversion with MLlib
is proposed initially.
h4. Validation cost
An ARRAY-to-VECTOR cast must inspect each value to validate its dimension and
null coordinates. Readers can avoid repeated validation when a trusted physical
format provides a fixed-size representation with required coordinates.
h4. Evolving storage specifications
The Parquet and Iceberg proposals may change during implementation. Mitigation:
keep the Spark logical contract independent of a physical encoding and avoid a
Spark-private encoding.
h4. Old-reader compatibility
Older Spark versions and protocol clients will not recognize VectorType. The
implementation must define explicit compatibility behavior and must not
silently reinterpret an unsupported logical type.
h4. Scope expansion
Vector types are often discussed together with distance functions, ANN indexes,
tensors, and quantization. The initial work will remain limited to the type
system, validation, core interoperability, and persistence.
*Q7. How long will it take?*
The expected implementation time is approximately six to nine months, with work
proceeding in parallel and persistence depending on the corresponding
file-format specifications.
# Base type functionality - 4 to 6 weeks
## Add VectorType, SQL parser and DDL support.
## Add JSON and catalog serialization.
## Add Scala, Java, and PySpark type APIs.
# Casts and Catalyst integration - 4 to 6 weeks
## Add validated ARRAY-to-VECTOR casts and lossless VECTOR-to-ARRAY conversion.
## Define common-type and expression behavior.
## Integrate interpreted and code-generated execution.
## Add structured errors.
# Persistence and Arrow integration - 2 to 3 months
## Add the standardized Parquet mapping.
## Add Arrow FixedSizeList conversion.
## Add columnar reader/writer and data-source conversion support.
## Define behavior for unsupported formats.
# Client support - 1 to 2 months
## Spark Connect protocol and clients.
## PySpark, JDBC, and Hive/Thrift metadata.
## Pandas and Arrow UDF paths where applicable.
# Testing, documentation, compatibility, and benchmarks - 1 month
Vector functions, indexes, MLlib conversions, and low-precision coordinate
types will be follow-up issues.
*Q8. What are the mid-term and final "exams" to check for success?*
h4. Mid-term criteria
* Users can declare {{VECTOR(N, T)}} in SQL and construct the equivalent type
in Scala, Java, and PySpark.
* The type survives schema JSON and catalog serialization.
* ARRAY-to-VECTOR casts reject wrong dimensions and null coordinates with
structured errors.
* Execution reuses ArrayData.
* Projection, filtering, shuffle, caching, and columnar transitions preserve
the type and values.
* The distinction from MLlib VectorUDT is documented and tested.
h4. Final criteria
* Spark can read and write the standardized Parquet representation.
* Arrow conversion uses FixedSizeList.
* Spark Connect, PySpark, and JDBC preserve or report the coordinate type and
dimension correctly.
* Spark maps all four initial coordinate types to the corresponding Iceberg
vector types.
* Unsupported formats fail clearly or require an explicit cast to ARRAY.
* Existing ARRAY behavior and performance do not regress.
* Vector scans and projections perform comparably to equivalent numeric arrays.
* Cross-engine tests preserve element type, dimension, top-level nullability,
and non-null coordinate semantics.
*Appendix A. Proposed API Changes.*
h4. SQL type syntax
{code:sql}
VECTOR(<dimension>, <coordinate type>)
{code}
Examples:
{code:sql}
VECTOR(3, INT)
VECTOR(128, BIGINT)
VECTOR(768, FLOAT)
VECTOR(1536, DOUBLE)
{code}
No default coordinate type is proposed. Both parameters are required.
h4. Scala API
Conceptually:
{code:scala}
package org.apache.spark.sql.types
case class VectorType(
elementType: DataType,
dimension: Int) extends DataType
{code}
Example:
{code:scala}
val schema = StructType(Seq(
StructField("embedding", VectorType(FloatType, 768), nullable = true)
))
{code}
h4. Java API
{code:java}
DataType vectorType =
DataTypes.createVectorType(DataTypes.FloatType, 768);
{code}
h4. PySpark API
{code:python}
from pyspark.sql.types import FloatType, StructField, StructType, VectorType
schema = StructType([
StructField("embedding", VectorType(FloatType(), 768), nullable=True)
])
{code}
Values use the same external representation as the corresponding numeric array.
No new user-facing vector value class is required initially.
h4. Type serialization
Proposed JSON:
{code:json}
{
"type": "vector",
"elementType": "float",
"dimension": 768
}
{code}
Proposed canonical type string: {{vector(768,float)}}.
*Appendix B. Type System.*
h4. Type identity
Both coordinate type and dimension are part of type identity:
{code}
VECTOR(768, FLOAT) != VECTOR(1536, FLOAT)
VECTOR(768, FLOAT) != VECTOR(768, DOUBLE)
{code}
h4. Coordinate types
||SQL spelling||Spark type||
|INT or INTEGER|IntegerType|
|BIGINT|LongType|
|FLOAT or REAL|FloatType|
|DOUBLE or DOUBLE PRECISION|DoubleType|
DECIMAL, strings, booleans, intervals, and nested types are not supported
initially.
h4. Nullability
VectorType has no {{containsNull}} parameter. Coordinate nullability is always
false. Top-level nullability remains a property of StructField.
h4. Floating-point special values
NaN and positive or negative infinity remain representable, following FloatType
and DoubleType. Individual functions, writers, indexes, or table constraints
may impose a finite-value requirement.
h4. Casting
||Source||Target||Behavior||
|ARRAY<T>|VECTOR(N,T)|Explicit; validates dimension and null coordinates|
|ARRAY<S>|VECTOR(N,T)|Explicit; applies normal numeric casting and vector
validation|
|VECTOR(N,T)|ARRAY<T>|Lossless|
|VECTOR(N,S)|VECTOR(N,T)|Explicit numeric cast|
|VECTOR(N,T)|VECTOR(M,T), N != M|Rejected at analysis time|
No implicit vector-to-vector coordinate promotion is proposed initially.
h4. Comparison
Equality and hashing use element-by-element semantics consistent with the
equivalent Spark array. Ordering comparisons and ORDER BY on a vector are
unsupported initially. Vector columns are not supported as table partition
columns, bucket columns, map keys, or range-clustering keys initially.
h4. Schema evolution
Changing the dimension is incompatible. Changing the coordinate type is also
incompatible as an in-place schema evolution operation initially, even where an
explicit value-level cast exists. Top-level nullability follows existing Spark
field-nullability rules.
*Appendix C. Internal and Storage Representations.*
h4. In-memory representation
A vector value uses ArrayData and the existing primitive coordinate
representation. Optimizer, UnsafeRow, shuffle, cache, and columnar execution
paths can reuse existing array machinery while retaining VectorType in the
logical schema.
h4. Arrow
The canonical Arrow mapping is {{VectorType(T, N) <-> FixedSizeList<T, N>}}
with a non-nullable child field.
h4. Parquet
The canonical mapping will follow the accepted Parquet fixed-size-list/vector
representation. It must preserve dimension, coordinate type, and required
coordinates. An ordinary LIST is not inferred to be a vector; legacy data
requires an explicit validated cast. Spark will not introduce a private Parquet
annotation as part of this SPIP.
h4. Iceberg
Expected mapping:
{code}
VectorType(IntegerType, N) -> int[N]
VectorType(LongType, N) -> long[N]
VectorType(FloatType, N) -> float[N]
VectorType(DoubleType, N) -> double[N]
{code}
The final serialized spelling is controlled by the Iceberg specification.
h4. Spark Connect
Conceptually, Spark Connect will add:
{code:protobuf}
message Vector {
DataType element_type = 1;
int32 dimension = 2;
}
{code}
*Appendix D. Relationship to MLlib.*
The new type does not replace or change VectorUDT. A future follow-up may add
explicit conversion between a dense MLlib Double vector and {{VECTOR(N,
DOUBLE)}}. Conversion from a sparse MLlib vector requires explicit
densification and is outside this proposal.
*Appendix E. Alternatives Considered.*
h4. Continue using ARRAY
Rejected because an array does not declare a dimension or vector identity.
Metadata and checks are not portable enough for catalogs, clients, file
formats, and function resolution.
h4. Add a FLOAT-only vector
Rejected because it prevents lossless interoperability with Double-valued
numerical systems and does not align with the initial Iceberg numeric-vector
proposal. FLOAT remains the primary expected embedding type.
h4. Reuse MLlib VectorUDT
Rejected because it is Double-only, supports sparse and variable-dimension
values, has a UDT struct encoding, and is not a native SQL or open-format
vector type.
h4. Add a generic FIXED_SIZE_ARRAY<T,N>
Rejected for the initial proposal because it introduces a broader type without
a demonstrated non-numeric use case. A generic fixed-size container could be
proposed independently later.
h4. Add a specialized packed physical representation
Rejected initially because it would require new row, columnar, shuffle,
code-generation, and storage paths. Reusing ArrayData provides the logical
benefits with substantially less risk.
h4. Include FP16, BF16, and INT8 initially
Deferred because Spark and Iceberg do not currently expose all of them as
ordinary primitive types. FP16 is a strong future coordinate type; BF16 is
useful for model interoperability; INT8 is useful for quantized search but may
require additional quantization metadata.
*References*
* [SQL:202y vector proposal
overview|https://peter.eisentraut.org/blog/2025/06/24/waiting-for-sql-202y-vectors]
* [Iceberg Vector Type Support
Proposal|https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit]
* [Parquet fixed-size vector/list
proposal|https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit]
* [Apache Arrow Fixed-Size List
layout|https://arrow.apache.org/docs/format/Columnar.html#fixed-size-list-layout]
* [Spark MLlib Vector
API|https://spark.apache.org/docs/latest/api/java/org/apache/spark/ml/linalg/Vector.html]
* [BigQuery vector
search|https://cloud.google.com/bigquery/docs/reference/standard-sql/search_functions#vector_search]
* [SPARK-51162: Add the TIME data
type|https://issues.apache.org/jira/browse/SPARK-51162]
* [SPARK-51658: Add geospatial types in
Spark|https://issues.apache.org/jira/browse/SPARK-51658]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
