[
https://issues.apache.org/jira/browse/SPARK-61228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruifeng Zheng updated SPARK-61228:
----------------------------------
Description:
The current vector_to_array implementation uses Scala UDFs that deserialize
vector objects. Add a dedicated Catalyst ML expression, VectorToArray, to
convert vector SQL structs directly to arrays of doubles or floats in
interpreted and generated execution.
The expression accepts only the exact vector SQL struct schema, including field
nullability and containsNull=false for the indices and values arrays. Use
VectorToArray(UnwrapUDT(input), targetType) in ML InternalFunctionRegistration.
Do not add a VectorToArray entry to Catalyst FunctionRegistry.scala.
Cover both ML and MLlib vector UDTs through the ML API, explicit unwrapping,
dense/sparse/empty vectors, both output types, and schema nullability checks.
Pull request: https://github.com/apache/spark/pull/60224
was:
Add a dedicated Catalyst ML expression, VectorToArray, that converts a vector
SQL struct to an array of doubles or floats. Register ml_vector_to_array in the
internal FunctionRegistry.
The expression accepts only the exact vector SQL struct schema, including
struct-field nullability and array-element containsNull flags. UDT columns must
first be converted with unwrap_udt. Interpreted and generated evaluation work
directly on Catalyst rows and arrays.
This change only adds the internal expression.
InternalFunctionRegistration.scala and the public vector_to_array UDF
implementation remain unchanged; public integration is deferred.
Pull request: https://github.com/apache/spark/pull/60224
> [SQL][ML] Add a dedicated vector_to_array Catalyst expression
> -------------------------------------------------------------
>
> Key: SPARK-61228
> URL: https://issues.apache.org/jira/browse/SPARK-61228
> Project: Spark
> Issue Type: Improvement
> Components: ML, SQL
> Affects Versions: 5.0.0
> Reporter: Ruifeng Zheng
> Priority: Major
> Labels: pull-request-available
>
> The current vector_to_array implementation uses Scala UDFs that deserialize
> vector objects. Add a dedicated Catalyst ML expression, VectorToArray, to
> convert vector SQL structs directly to arrays of doubles or floats in
> interpreted and generated execution.
> The expression accepts only the exact vector SQL struct schema, including
> field nullability and containsNull=false for the indices and values arrays.
> Use VectorToArray(UnwrapUDT(input), targetType) in ML
> InternalFunctionRegistration. Do not add a VectorToArray entry to Catalyst
> FunctionRegistry.scala.
> Cover both ML and MLlib vector UDTs through the ML API, explicit unwrapping,
> dense/sparse/empty vectors, both output types, and schema nullability checks.
> Pull request: https://github.com/apache/spark/pull/60224
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
