This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 22dae1026648 [MINOR][ML][TEST] Add tests for vector UDFs with null
inputs
22dae1026648 is described below
commit 22dae1026648b3b24ea32ac94b841585a98266ab
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Fri Aug 28 23:51:08 2026 +0800
[MINOR][ML][TEST] Add tests for vector UDFs with null inputs
### What changes were proposed in this pull request?
Add ML test coverage for Scala UDFs applied to datasets containing dense,
sparse, and null vectors.
The test verifies constant-returning and null-checking UDF behavior, and
verifies that dereferencing
a null vector reports `FAILED_EXECUTE_UDF` with a `NullPointerException`
cause.
### Why are the changes needed?
The tests document the null-input behavior of Scala UDFs whose input is an
ML `Vector`.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
`JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 build/sbt 'mllib/testOnly
org.apache.spark.ml.util.MLTestSuite'`
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
Closes #58382 from zhengruifeng/ml-udf-null-vector-dev-5.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit 7e8dc46270cfa2f2ebc9dfece8820cb0b1caaad8)
Signed-off-by: Ruifeng Zheng <[email protected]>
---
.../org/apache/spark/ml/util/MLTestSuite.scala | 22 ++++++++++++++++++++++
1 file changed, 22 insertions(+)
diff --git a/mllib/src/test/scala/org/apache/spark/ml/util/MLTestSuite.scala
b/mllib/src/test/scala/org/apache/spark/ml/util/MLTestSuite.scala
index 1732469ccf59..ad36d253a996 100644
--- a/mllib/src/test/scala/org/apache/spark/ml/util/MLTestSuite.scala
+++ b/mllib/src/test/scala/org/apache/spark/ml/util/MLTestSuite.scala
@@ -17,13 +17,35 @@
package org.apache.spark.ml.util
+import org.apache.spark.SparkException
import org.apache.spark.ml.feature.StringIndexer
+import org.apache.spark.ml.linalg.{Vector, Vectors}
import org.apache.spark.sql.Row
+import org.apache.spark.sql.functions.udf
class MLTestSuite extends MLTest {
import testImplicits._
+ test("UDF with vector input containing null") {
+ val data = Seq(
+ Tuple1(Vectors.dense(1.0, 2.0)),
+ Tuple1(Vectors.sparse(2, Seq(0 -> 1.0))),
+ Tuple1(null.asInstanceOf[Vector])).toDF("vec")
+ val constantUDF = udf { vec: Vector => 1.0 }
+ val isNullUDF = udf { vec: Vector => vec == null }
+ val sizeUDF = udf { vec: Vector => vec.size }
+
+ checkAnswer(data.select(constantUDF($"vec")), Seq(Row(1.0), Row(1.0),
Row(1.0)))
+ checkAnswer(data.select(isNullUDF($"vec")), Seq(Row(false), Row(false),
Row(true)))
+
+ val exception = intercept[SparkException] {
+ data.select(sizeUDF($"vec")).collect()
+ }
+ assert(exception.getCondition === "FAILED_EXECUTE_UDF")
+ assert(exception.getCause.isInstanceOf[NullPointerException])
+ }
+
test("test transformer on stream data") {
val data = Seq((0, "a"), (1, "b"), (2, "c"), (3, "d"), (4, "e"), (5, "f"))
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
