[
https://issues.apache.org/jira/browse/SPARK-61228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruifeng Zheng updated SPARK-61228:
----------------------------------
Description:
Add a dedicated Catalyst ML expression, VectorToArray, that converts a vector
SQL struct to an array of doubles or floats. Register ml_vector_to_array in the
internal FunctionRegistry.
The expression accepts only the exact vector SQL struct schema, including
struct-field nullability and array-element containsNull flags. UDT columns must
first be converted with unwrap_udt. Interpreted and generated evaluation work
directly on Catalyst rows and arrays.
This change only adds the internal expression.
InternalFunctionRegistration.scala and the public vector_to_array UDF
implementation remain unchanged; public integration is deferred.
Pull request: https://github.com/apache/spark/pull/60224
was:
The current vector_to_array implementation uses Scala UDFs that deserialize
vector objects. Add a dedicated Catalyst ML expression that accepts an input
expression and a target datatype (double or float), and accepts both a vector
SQL struct and any UDT whose underlying SQL type matches the vector schema.
Use the expression in InternalFunctionRegistration and register
ml_vector_to_array in FunctionRegistry. Evaluate directly on Catalyst rows and
arrays in interpreted and generated execution, with coverage for dense, sparse,
empty, and matching-UDT inputs.
Pull request: https://github.com/apache/spark/pull/60224
> [SQL][ML] Add a dedicated vector_to_array Catalyst expression
> -------------------------------------------------------------
>
> Key: SPARK-61228
> URL: https://issues.apache.org/jira/browse/SPARK-61228
> Project: Spark
> Issue Type: Improvement
> Components: ML, SQL
> Affects Versions: 5.0.0
> Reporter: Ruifeng Zheng
> Priority: Major
> Labels: pull-request-available
>
> Add a dedicated Catalyst ML expression, VectorToArray, that converts a vector
> SQL struct to an array of doubles or floats. Register ml_vector_to_array in
> the internal FunctionRegistry.
> The expression accepts only the exact vector SQL struct schema, including
> struct-field nullability and array-element containsNull flags. UDT columns
> must first be converted with unwrap_udt. Interpreted and generated evaluation
> work directly on Catalyst rows and arrays.
> This change only adds the internal expression.
> InternalFunctionRegistration.scala and the public vector_to_array UDF
> implementation remain unchanged; public integration is deferred.
> Pull request: https://github.com/apache/spark/pull/60224
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
