This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new f33f880cb3f1 [SPARK-58692][ML] Optimize PolynomialExpansion transform
closure
f33f880cb3f1 is described below
commit f33f880cb3f16996c80a48726f47cdaed6701784
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Tue Aug 11 08:08:38 2026 +0800
[SPARK-58692][ML] Optimize PolynomialExpansion transform closure
### What changes were proposed in this pull request?
Snapshot the configured polynomial degree while creating
PolynomialExpansion's transform function. The resulting function no longer
captures the PolynomialExpansion transformer.
The serialized transform function decreases from 3,086 bytes to 1,322 bytes
(57.2%).
### Why are the changes needed?
Avoiding the transformer capture reduces the memory retained by the
transform closure, which is especially useful for Spark Connect server
workloads.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Compiled the edited PolynomialExpansion source and serialized its generated
transform function with Java serialization. The existing
PolynomialExpansionSuite covers the unchanged transform behavior.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
Closes #57895 from
zhengruifeng/SPARK-58692-polynomial-expansion-transform-closure.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
---
.../main/scala/org/apache/spark/ml/feature/PolynomialExpansion.scala | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/feature/PolynomialExpansion.scala
b/mllib/src/main/scala/org/apache/spark/ml/feature/PolynomialExpansion.scala
index aa61b2335487..c89da4394ce8 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/feature/PolynomialExpansion.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/feature/PolynomialExpansion.scala
@@ -64,8 +64,9 @@ class PolynomialExpansion @Since("1.4.0") (@Since("1.4.0")
override val uid: Str
@Since("1.4.0")
def setDegree(value: Int): this.type = set(degree, value)
- override protected def createTransformFunc: Vector => Vector = { v =>
- PolynomialExpansion.expand(v, $(degree))
+ override protected def createTransformFunc: Vector => Vector = {
+ val localDegree = $(degree)
+ v => PolynomialExpansion.expand(v, localDegree)
}
override protected def validateInputType(inputType: DataType): Unit = {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
