[
https://issues.apache.org/jira/browse/SPARK-61518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruifeng Zheng updated SPARK-61518:
----------------------------------
Component/s: ML
Summary: [SQL][ML] Replace vector_to_array UDFs with a
runtime-replaceable expression (was: Replace vector_to_array UDFs with a
runtime-replaceable expression)
> [SQL][ML] Replace vector_to_array UDFs with a runtime-replaceable expression
> ----------------------------------------------------------------------------
>
> Key: SPARK-61518
> URL: https://issues.apache.org/jira/browse/SPARK-61518
> Project: Spark
> Issue Type: Improvement
> Components: ML, SQL
> Affects Versions: 5.0.0
> Reporter: Ruifeng Zheng
> Priority: Major
>
> vector_to_array currently deserializes both ML vector UDT versions into JVM
> objects and executes a Scala UDF for each output element type.
> Replace the conversion UDFs with a runtime-replaceable VectorToArray Catalyst
> expression over the canonical vector struct. The internal ML function builder
> unwraps the UDT with UnwrapUDT. The replacement casts the dense values array
> to the requested double/float array type or expands a sparse vector with the
> existing SparseVectorToArray expression. Preserve vector-schema checks,
> including non-null indices/values elements.
> Null vectors propagate null rather than throwing from the UDF, and
> incompatible inputs fail analysis. Both ML and legacy MLlib vector UDTs and
> the float64/float32 dtype options remain supported.
> Add Catalyst evaluation, schema, and UDT tests, and update the ML integration
> coverage to verify nulls, empty and special values, both UDTs and dtypes, and
> the absence of conversion UDFs in the optimized plan.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]