This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.3 by this push:
new 4a669e86a940 [SPARK-60461][MLLIB] Restore Strategy constructor binary
compatibility
4a669e86a940 is described below
commit 4a669e86a940d9d54896ba00dd5c7fb930a05551
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Fri Oct 23 08:21:05 2026 +0800
[SPARK-60461][MLLIB] Restore Strategy constructor binary compatibility
### What changes were proposed in this pull request?
Restore the 15-argument `Strategy` constructor used in Spark 4.2,
delegating to the primary constructor with pruning enabled. Keep `bootstrap` in
its original parameter position, after `minWeightFractionPerNode`, so its
constructor default getter continues to return `false`. Make `pruneTree`
`private[spark]`, preserve both `bootstrap` and `pruneTree` in `Strategy.copy`,
and remove the corresponding MiMa constructor exclusion.
Please backport to `branch-4.x` and `branch-4.3`.
### Why are the changes needed?
#55763 added `pruneTree` before `bootstrap`, changing the JVM constructor
from 15 to 16 parameters. Existing applications compiled against Spark 4.2 fail
with `NoSuchMethodError` on the current 4.3 branch. For example, a client
compiled against 4.2 with `new Strategy(Algo.Classification, Gini, maxDepth =
3)` fails at construction.
Adding only the old constructor would leave another compatibility problem:
the 15th constructor default getter would return the new pruning default
(`true`) instead of the old bootstrap default (`false`). Retaining the original
parameter positions preserves both linkage and behavior.
`private[spark] var bootstrap` restricts Scala access to the stored
property and its accessors. It does not restrict callers from supplying that
constructor argument or make the constructor package-private. Spark 4.2's
15-argument constructor was public, so the compatibility constructor must also
remain public.
For example, code in a package outside Spark can supply `bootstrap` either
by name, using defaults for the other optional arguments, or explicitly as the
last of all 15 arguments:
```scala
package example
import org.apache.spark.mllib.tree.configuration.{Algo, QuantileStrategy,
Strategy}
import org.apache.spark.mllib.tree.impurity.Gini
object StrategyClient {
val named = new Strategy(Algo.Classification, Gini, maxDepth = 3,
bootstrap = true)
val explicit = new Strategy(
Algo.Classification, Gini, 3, 2, 32, QuantileStrategy.Sort,
Map.empty[Int, Int], 1, 0.0, 256, 1.0, false, 10, 0.0, true)
}
```
Both calls compile against Spark 4.2; the final `true` in the second call
is the `bootstrap` argument. The compiler fills the omitted defaults in the
first call and invokes the same public 15-argument constructor.
Reading or writing the property afterward is still rejected in that
external package. These examples intentionally do not compile:
```scala
object InvalidAccess {
val bootstrap = StrategyClient.named.bootstrap
StrategyClient.named.bootstrap = false
}
```
The constructor examples compile against both Spark 4.2 and the patched
`Strategy`, while both property-access examples are rejected by the Scala
compiler in each version.
### Does this PR introduce _any_ user-facing change?
Yes. Applications compiled against Spark 4.2 can construct `Strategy` on
Spark 4.3 again, preserving the default and explicit bootstrap values. Pruning
remains enabled by default.
### How was this patch tested?
Added three regression tests in `DecisionTreeSuite` covering the old JVM
constructor signature, the bootstrap constructor default getter, and
independent pruning/bootstrapping configuration, including copying all four
combinations of the two flags. All three passed using the ScalaTest runner with
the `Strategy` test-name filter with locally cached Spark dependencies.
Compiled a separate Scala client against the published Spark 4.2.0 MLlib
artifact. It reproduces `NoSuchMethodError` with the unpatched 4.3 class and
runs successfully with the patched class, preserving the default bootstrap
value and both explicit boolean values. The changed class and regression suite
were compiled directly with Scala 2.13.18. Full SBT and MiMa runs were not
performed.
Audited the `Strategy` constructor/factory call sites, both training
copies, ML classifier parameter conversion, and recursive node conversion; no
further propagation changes were needed. Also ran the two existing pruning
regression tests and three temporary integration tests covering legacy
decision-tree/random-forest classification and regression, ML classifiers, and
gradient boosting with pruning enabled and disabled. All eight focused tests
passed.
Checked whitespace, ASCII, and changed-line lengths. The patch applies
cleanly to `branch-4.3`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6)
Closes #59704 from zhengruifeng/ml-strategy-constructor-compatibility.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit fce7d51a3760953097fe34df8aa9a4f5326c74ff)
Signed-off-by: Ruifeng Zheng <[email protected]>
---
.../spark/mllib/tree/configuration/Strategy.scala | 35 +++++++++++++++++---
.../spark/mllib/tree/DecisionTreeSuite.scala | 37 ++++++++++++++++++++++
project/MimaExcludes.scala | 2 --
3 files changed, 67 insertions(+), 7 deletions(-)
diff --git
a/mllib/src/main/scala/org/apache/spark/mllib/tree/configuration/Strategy.scala
b/mllib/src/main/scala/org/apache/spark/mllib/tree/configuration/Strategy.scala
index c1f82157e82f..0e8501df4a8f 100644
---
a/mllib/src/main/scala/org/apache/spark/mllib/tree/configuration/Strategy.scala
+++
b/mllib/src/main/scala/org/apache/spark/mllib/tree/configuration/Strategy.scala
@@ -65,8 +65,6 @@ import org.apache.spark.mllib.tree.impurity.{Entropy, Gini,
Impurity, Variance}
* E.g. 10 means that the cache will get
checkpointed every 10 updates. If
* the checkpoint directory is not set in
* [[org.apache.spark.SparkContext]], this setting
is ignored.
- * @param pruneTree If this is true, the final training tree will undergo a
pruning in which
- * nodes with the same prediction are merged.
*/
@Since("1.0.0")
class Strategy @Since("1.3.0") (
@@ -84,8 +82,8 @@ class Strategy @Since("1.3.0") (
@Since("1.2.0") @BeanProperty var useNodeIdCache: Boolean = false,
@Since("1.2.0") @BeanProperty var checkpointInterval: Int = 10,
@Since("3.0.0") @BeanProperty var minWeightFractionPerNode: Double = 0.0,
- @Since("4.3.0") @BeanProperty var pruneTree: Boolean = true,
- @BeanProperty private[spark] var bootstrap: Boolean = false) extends
Serializable {
+ @BeanProperty private[spark] var bootstrap: Boolean = false,
+ @BeanProperty private[spark] var pruneTree: Boolean = true) extends
Serializable {
/**
*/
@@ -102,6 +100,31 @@ class Strategy @Since("1.3.0") (
}
// scalastyle:off argcount
+ /**
+ * Retains the constructor signature used before the pruning parameter was
introduced.
+ */
+ def this(
+ algo: Algo,
+ impurity: Impurity,
+ maxDepth: Int,
+ numClasses: Int,
+ maxBins: Int,
+ quantileCalculationStrategy: QuantileStrategy,
+ categoricalFeaturesInfo: Map[Int, Int],
+ minInstancesPerNode: Int,
+ minInfoGain: Double,
+ maxMemoryInMB: Int,
+ subsamplingRate: Double,
+ useNodeIdCache: Boolean,
+ checkpointInterval: Int,
+ minWeightFractionPerNode: Double,
+ bootstrap: Boolean) = {
+ this(algo, impurity, maxDepth, numClasses, maxBins,
quantileCalculationStrategy,
+ categoricalFeaturesInfo, minInstancesPerNode, minInfoGain, maxMemoryInMB,
+ subsamplingRate, useNodeIdCache, checkpointInterval,
minWeightFractionPerNode,
+ bootstrap, pruneTree = true)
+ }
+
/**
* Backwards compatible constructor for
[[org.apache.spark.mllib.tree.configuration.Strategy]]
*/
@@ -204,7 +227,9 @@ class Strategy @Since("1.3.0") (
new Strategy(algo, impurity, maxDepth, numClasses, maxBins,
quantileCalculationStrategy, categoricalFeaturesInfo,
minInstancesPerNode,
minInfoGain, maxMemoryInMB, subsamplingRate, useNodeIdCache,
- checkpointInterval, minWeightFractionPerNode, pruneTree)
+ checkpointInterval, minWeightFractionPerNode,
+ bootstrap = bootstrap,
+ pruneTree = pruneTree)
}
}
diff --git
a/mllib/src/test/scala/org/apache/spark/mllib/tree/DecisionTreeSuite.scala
b/mllib/src/test/scala/org/apache/spark/mllib/tree/DecisionTreeSuite.scala
index 2f5579acea46..488f7b9e200e 100644
--- a/mllib/src/test/scala/org/apache/spark/mllib/tree/DecisionTreeSuite.scala
+++ b/mllib/src/test/scala/org/apache/spark/mllib/tree/DecisionTreeSuite.scala
@@ -25,6 +25,7 @@ import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.mllib.tree.configuration.Algo._
import org.apache.spark.mllib.tree.configuration.FeatureType._
+import org.apache.spark.mllib.tree.configuration.QuantileStrategy.Sort
import org.apache.spark.mllib.tree.configuration.Strategy
import org.apache.spark.mllib.tree.impurity.{Entropy, Gini, Variance}
import org.apache.spark.mllib.tree.model._
@@ -35,6 +36,42 @@ import org.apache.spark.util.Utils
class DecisionTreeSuite extends SparkFunSuite with MLlibTestSparkContext {
+ /////////////////////////////////////////////////////////////////////////////
+ // Tests for Strategy configuration
+ /////////////////////////////////////////////////////////////////////////////
+
+ test("Strategy should retain the constructor signature used by Spark 4.2") {
+ assert(classOf[Strategy].getConstructors.exists(_.getParameterCount == 15))
+ for (bootstrap <- Seq(false, true)) {
+ val strategy = new Strategy(Classification, Gini, 3, 2, 32, Sort,
Map.empty[Int, Int],
+ 1, 0.0, 256, 1.0, false, 10, 0.0, bootstrap)
+
+ assert(strategy.bootstrap == bootstrap)
+ assert(strategy.pruneTree)
+ }
+ }
+
+ test("Strategy should retain the bootstrap default getter used by compiled
callers") {
+ val bootstrapDefault =
Strategy.getClass.getMethod("$lessinit$greater$default$15")
+ assert(bootstrapDefault.invoke(Strategy) == java.lang.Boolean.FALSE)
+ }
+
+ test("Strategy should configure pruning independently of bootstrapping") {
+ for {
+ bootstrap <- Seq(false, true)
+ pruneTree <- Seq(false, true)
+ } {
+ val strategy = new Strategy(Classification, Gini, 3,
+ bootstrap = bootstrap, pruneTree = pruneTree)
+ assert(strategy.bootstrap == bootstrap)
+ assert(strategy.pruneTree == pruneTree)
+
+ val copied = strategy.copy
+ assert(copied.bootstrap == bootstrap)
+ assert(copied.pruneTree == pruneTree)
+ }
+ }
+
/////////////////////////////////////////////////////////////////////////////
// Tests calling train()
/////////////////////////////////////////////////////////////////////////////
diff --git a/project/MimaExcludes.scala b/project/MimaExcludes.scala
index c99a72a1e2a8..37139b23c56f 100644
--- a/project/MimaExcludes.scala
+++ b/project/MimaExcludes.scala
@@ -98,8 +98,6 @@ object MimaExcludes {
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.addTaskInterruptListener"),
// [SPARK-56700][SS] Make DataStreamReader.name public
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.streaming.DataStreamReader.name"),
- // [SPARK-34591][ML] Add pruneTree parameter to Strategy
-
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.mllib.tree.configuration.Strategy.this"),
// [SPARK-56395][SQL] Add NEAREST BY top-K ranking join
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.Dataset.nearestByJoin"),
// [SPARK-57332][SQL] MySQLDialect no longer overrides the visit methods
below; the public
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]