This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 42afafaf7b88 [SPARK-60608][ML][CONNECT][TEST][FOLLOWUP] Strengthen
nested model allowlist regression coverage
42afafaf7b88 is described below
commit 42afafaf7b880364013a5384da4497eff5ee0ec7
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Sat Nov 7 19:10:12 2026 +0800
[SPARK-60608][ML][CONNECT][TEST][FOLLOWUP] Strengthen nested model
allowlist regression coverage
### What changes were proposed in this pull request?
Replace the unregistered stage fixture in MLClassLoadingSuite with a custom
MLReader that reads its saved UID directly from metadata. Record calls to the
fixture's read method and assert that classic loading invokes it while Spark
Connect rejects the nested stage before invoking it. Keep the existing
allowlisted and fitted RFormulaModel round trips.
### Why are the changes needed?
The previous fixture used DefaultParamsReadable, whose DefaultParamsReader
already checked the allowlist. Its rejection test therefore passed even without
the nested reader check added by SPARK-60608. The custom reader and call
assertions cover that check directly.
### Does this PR introduce _any_ user-facing change?
No. This changes tests only.
### How was this patch tested?
Scala parsing and formatting with the repository's Scalafmt 3.8.6
configuration, ASCII and line-length checks, and git diff --check passed.
The focused Spark SBT test stopped before suite compilation. Default
repositories failed DNS resolution. Using the Maven mirror then hit a forbidden
snowflake-jdbc-thin 4.4.1 download and the branch's requirement for Java 25.0.3
or later; the installed JDK is 25.0.2.
The same test source was ported to Runtime and its focused
MLClassLoadingSuite passed all four tests with Bazel. Only whitespace
formatting changed after that run.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-6)
Closes #60170 from zhengruifeng/ml-connect-nested-reader-regression.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit dad84ea00dee7d1717fe8638eecb801486c9b3ce)
Signed-off-by: Ruifeng Zheng <[email protected]>
skip-checks: true
---
.../spark/sql/connect/ml/MLClassLoadingSuite.scala | 29 +++++++++++++++++++---
1 file changed, 26 insertions(+), 3 deletions(-)
diff --git
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/ml/MLClassLoadingSuite.scala
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/ml/MLClassLoadingSuite.scala
index d51997bad213..09dd071200b6 100644
---
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/ml/MLClassLoadingSuite.scala
+++
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/ml/MLClassLoadingSuite.scala
@@ -18,12 +18,13 @@
package org.apache.spark.sql.connect.ml
import java.io.File
+import java.util.concurrent.atomic.AtomicInteger
import org.apache.spark.connect.proto
import org.apache.spark.ml.{Model, Transformer}
import org.apache.spark.ml.feature.{RFormula, RFormulaModel,
RFormulaModelTestHelper, VectorAssembler}
import org.apache.spark.ml.param.ParamMap
-import org.apache.spark.ml.util.{DefaultParamsReadable, DefaultParamsWritable}
+import org.apache.spark.ml.util.{DefaultParamsWritable, MLReadable, MLReader}
import org.apache.spark.sql.{DataFrame, Dataset}
import org.apache.spark.sql.connect.SparkConnectTestUtils
import org.apache.spark.sql.types.StructType
@@ -40,7 +41,25 @@ class UnregisteredModel(override val uid: String)
override def copy(extra: ParamMap): UnregisteredModel = defaultCopy(extra)
}
-object UnregisteredModel extends DefaultParamsReadable[UnregisteredModel]
+object UnregisteredModel extends MLReadable[UnregisteredModel] {
+
+ val readCalls = new AtomicInteger()
+
+ override def read: MLReader[UnregisteredModel] = {
+ readCalls.incrementAndGet()
+ new MLReader[UnregisteredModel] {
+ override def load(path: String): UnregisteredModel = {
+ // Read metadata directly so DefaultParamsReader's allowlist check
cannot mask a regression.
+ val uid = sparkSession.read
+ .json(new File(path, "metadata").getAbsolutePath)
+ .select("uid")
+ .head()
+ .getString(0)
+ new UnregisteredModel(uid)
+ }
+ }
+ }
+}
class MLClassLoadingSuite extends MLHelper {
@@ -116,20 +135,24 @@ class MLClassLoadingSuite extends MLHelper {
withSavedModel(stage) { path =>
val nestedPath = new File(path,
s"pipelineModel/stages/0_${stage.uid}").getAbsolutePath
MLHandler.currentSessionHolder.remove()
+ val readCallsBefore = UnregisteredModel.readCalls.get()
val model = UnregisteredModel.load(nestedPath)
assert(model.uid == stage.uid)
+ assert(UnregisteredModel.readCalls.get() == readCallsBefore + 1)
val transformers = MLUtils.loadOperators(classOf[Transformer])
assert(!transformers.contains(classOf[UnregisteredModel].getName))
}
}
- test("reject an RFormulaModel with a non-allowlisted nested stage") {
+ test("reject a non-allowlisted nested stage before invoking its reader") {
withSavedModel(new UnregisteredModel("stageFixture")) { path =>
+ val readCallsBefore = UnregisteredModel.readCalls.get()
val error = intercept[MlUnsupportedException] {
readModel(path)
}
assert(error.getMessage.contains(classOf[UnregisteredModel].getName))
assert(error.getMessage.contains("not in the allowlist"))
+ assert(UnregisteredModel.readCalls.get() == readCallsBefore)
}
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]