[
https://issues.apache.org/jira/browse/SPARK-61228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruifeng Zheng updated SPARK-61228:
----------------------------------
Description:
Add SparseVectorToArray, an internal Catalyst ML expression that expands a
sparse vector's size, indices, and double values into a dense array of doubles
or floats.
Require integer indices and double values with containsNull=false for both
arrays. Fill missing positions with zero and propagate null inputs. Implement
interpreted and generated evaluation, allocating the requested output type
directly.
This change adds only the sparse expansion expression and its tests. The
existing vector_to_array function and its registration remain unchanged.
Pull request: https://github.com/apache/spark/pull/60224
was:
The current vector_to_array implementation uses Scala UDFs that deserialize
each input vector. Add a dedicated Catalyst ML expression, VectorToArray, to
convert underlying vector rows to arrays of doubles or floats in interpreted
and generated execution.
Accept vector SQL structs and UDTs whose underlying SQL type matches the vector
schema, including field nullability and containsNull=false for the indices and
values arrays. Use VectorToArray(input, targetType) directly in ML
InternalFunctionRegistration for the existing vector_to_array function. The
expression also uses vector_to_array as its display name.
Cover ML and MLlib vector UDTs, SQL struct inputs, dense/sparse/empty vectors,
both output types, schema nullability, and the registered function's output
name.
Pull request: https://github.com/apache/spark/pull/60224
Summary: [SQL][ML] Add a SparseVectorToArray Catalyst expression (was:
[SQL][ML] Add a dedicated vector_to_array Catalyst expression)
> [SQL][ML] Add a SparseVectorToArray Catalyst expression
> -------------------------------------------------------
>
> Key: SPARK-61228
> URL: https://issues.apache.org/jira/browse/SPARK-61228
> Project: Spark
> Issue Type: Improvement
> Components: ML, SQL
> Affects Versions: 5.0.0
> Reporter: Ruifeng Zheng
> Priority: Major
> Labels: pull-request-available
>
> Add SparseVectorToArray, an internal Catalyst ML expression that expands a
> sparse vector's size, indices, and double values into a dense array of
> doubles or floats.
> Require integer indices and double values with containsNull=false for both
> arrays. Fill missing positions with zero and propagate null inputs. Implement
> interpreted and generated evaluation, allocating the requested output type
> directly.
> This change adds only the sparse expansion expression and its tests. The
> existing vector_to_array function and its registration remain unchanged.
> Pull request: https://github.com/apache/spark/pull/60224
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]