Ruifeng Zheng created SPARK-61518:
-------------------------------------
Summary: Replace vector_to_array UDFs with a runtime-replaceable
expression
Key: SPARK-61518
URL: https://issues.apache.org/jira/browse/SPARK-61518
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Ruifeng Zheng
vector_to_array currently deserializes both ML vector UDT versions into JVM
objects and executes a Scala UDF for each output element type.
Replace the conversion UDFs with a runtime-replaceable VectorToArray Catalyst
expression over the canonical vector struct. The internal ML function builder
unwraps the UDT with UnwrapUDT. The replacement casts the dense values array to
the requested double/float array type or expands a sparse vector with the
existing SparseVectorToArray expression. Preserve vector-schema checks,
including non-null indices/values elements.
Null vectors propagate null rather than throwing from the UDF, and incompatible
inputs fail analysis. Both ML and legacy MLlib vector UDTs and the
float64/float32 dtype options remain supported.
Add Catalyst evaluation, schema, and UDT tests, and update the ML integration
coverage to verify nulls, empty and special values, both UDTs and dtypes, and
the absence of conversion UDFs in the optimized plan.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]