[
https://issues.apache.org/jira/browse/SPARK-61228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruifeng Zheng updated SPARK-61228:
----------------------------------
Description:
The current vector_to_array implementation uses Scala UDFs that deserialize
each input vector. Add a dedicated Catalyst ML expression, VectorToArray, to
convert underlying vector rows to arrays of doubles or floats in interpreted
and generated execution.
Accept vector SQL structs and UDTs whose underlying SQL type matches the vector
schema, including field nullability and containsNull=false for the indices and
values arrays. Use VectorToArray(input, targetType) directly in ML
InternalFunctionRegistration for the existing vector_to_array function. The
expression also uses vector_to_array as its display name.
Cover ML and MLlib vector UDTs, SQL struct inputs, dense/sparse/empty vectors,
both output types, schema nullability, and the registered function's output
name.
Pull request: https://github.com/apache/spark/pull/60224
was:
The current vector_to_array implementation uses Scala UDFs that deserialize
vector objects. Add a dedicated Catalyst ML expression, VectorToArray, to
convert vector SQL structs directly to arrays of doubles or floats in
interpreted and generated execution.
The expression accepts only the exact vector SQL struct schema, including field
nullability and containsNull=false for the indices and values arrays. Use
VectorToArray(UnwrapUDT(input), targetType) in ML InternalFunctionRegistration.
Do not add a VectorToArray entry to Catalyst FunctionRegistry.scala.
Cover both ML and MLlib vector UDTs through the ML API, explicit unwrapping,
dense/sparse/empty vectors, both output types, and schema nullability checks.
Pull request: https://github.com/apache/spark/pull/60224
> [SQL][ML] Add a dedicated vector_to_array Catalyst expression
> -------------------------------------------------------------
>
> Key: SPARK-61228
> URL: https://issues.apache.org/jira/browse/SPARK-61228
> Project: Spark
> Issue Type: Improvement
> Components: ML, SQL
> Affects Versions: 5.0.0
> Reporter: Ruifeng Zheng
> Priority: Major
> Labels: pull-request-available
>
> The current vector_to_array implementation uses Scala UDFs that deserialize
> each input vector. Add a dedicated Catalyst ML expression, VectorToArray, to
> convert underlying vector rows to arrays of doubles or floats in interpreted
> and generated execution.
> Accept vector SQL structs and UDTs whose underlying SQL type matches the
> vector schema, including field nullability and containsNull=false for the
> indices and values arrays. Use VectorToArray(input, targetType) directly in
> ML InternalFunctionRegistration for the existing vector_to_array function.
> The expression also uses vector_to_array as its display name.
> Cover ML and MLlib vector UDTs, SQL struct inputs, dense/sparse/empty
> vectors, both output types, schema nullability, and the registered function's
> output name.
> Pull request: https://github.com/apache/spark/pull/60224
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
