[ 
https://issues.apache.org/jira/browse/SPARK-61518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ruifeng Zheng updated SPARK-61518:
----------------------------------
    Component/s: ML
        Summary: [SQL][ML] Replace vector_to_array UDFs with a 
runtime-replaceable expression  (was: Replace vector_to_array UDFs with a 
runtime-replaceable expression)

> [SQL][ML] Replace vector_to_array UDFs with a runtime-replaceable expression
> ----------------------------------------------------------------------------
>
>                 Key: SPARK-61518
>                 URL: https://issues.apache.org/jira/browse/SPARK-61518
>             Project: Spark
>          Issue Type: Improvement
>          Components: ML, SQL
>    Affects Versions: 5.0.0
>            Reporter: Ruifeng Zheng
>            Priority: Major
>
> vector_to_array currently deserializes both ML vector UDT versions into JVM 
> objects and executes a Scala UDF for each output element type.
> Replace the conversion UDFs with a runtime-replaceable VectorToArray Catalyst 
> expression over the canonical vector struct. The internal ML function builder 
> unwraps the UDT with UnwrapUDT. The replacement casts the dense values array 
> to the requested double/float array type or expands a sparse vector with the 
> existing SparseVectorToArray expression. Preserve vector-schema checks, 
> including non-null indices/values elements.
> Null vectors propagate null rather than throwing from the UDF, and 
> incompatible inputs fail analysis. Both ML and legacy MLlib vector UDTs and 
> the float64/float32 dtype options remain supported.
> Add Catalyst evaluation, schema, and UDT tests, and update the ML integration 
> coverage to verify nulls, empty and special values, both UDTs and dtypes, and 
> the absence of conversion UDFs in the optimized plan.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to