zhengruifeng opened a new pull request, #58124: URL: https://github.com/apache/spark/pull/58124
### What changes were proposed in this pull request? This PR adds a Catalyst `WrapUDT` expression that wraps a column with a target UDT when the column data type matches the UDT's underlying SQL type. It also updates `MLUtils.convertVectorColumnsToML` and `MLUtils.convertVectorColumnsFromML` to use `WrapUDT(UnwrapUDT(...))` for `.ml` / `.mllib` vector UDT conversion instead of Scala UDFs. The matrix conversion paths are unchanged. ### Why are the changes needed? `.ml` and `.mllib` vector UDTs use the same Catalyst storage layout, but the current conversion helpers deserialize each vector and create a new vector object through `asML` / `Vectors.fromML`. `WrapUDT` allows these helpers to change the logical UDT of a column without copying the underlying Catalyst value, avoiding per-row UDF conversion overhead for vector columns. ### Does this PR introduce _any_ user-facing change? No. Existing vector conversion helper APIs keep the same behavior and return schemas, but use a more efficient internal implementation. ### How was this patch tested? Added tests for `WrapUDT` positive and negative type checking. Updated `MLUtils` vector conversion tests to verify converted DataFrames use `WrapUDT` and do not use `ScalaUDF`. Ran: ``` build/sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 catalyst/Test/compile mllib/Test/compile ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
