This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new c32a77ae17a4 [SPARK-58688][ML] Optimize DCT transform closure
c32a77ae17a4 is described below
commit c32a77ae17a4120f45925e7524eca987b07f9ff3
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Mon Aug 10 20:12:39 2026 +0800
[SPARK-58688][ML] Optimize DCT transform closure
### What changes were proposed in this pull request?
Select a dedicated forward or inverse transform function from the
configured inverse-DCT flag. The resulting function captures neither the DCT
transformer nor the flag.
The serialized inverse transform function decreases from 2,863 bytes to
1,094 bytes (61.8%).
### Why are the changes needed?
Avoiding the transformer capture reduces the memory retained by the
transform closure, which is especially useful for Spark Connect server
workloads.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Compiled the edited DCT source and serialized its generated transform
function with Java serialization. The existing DCTSuite covers the unchanged
transform behavior.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
Closes #57889 from zhengruifeng/SPARK-58688-dct-transform-state.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
---
.../scala/org/apache/spark/ml/feature/DCT.scala | 22 +++++++++++++++++-----
1 file changed, 17 insertions(+), 5 deletions(-)
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/DCT.scala
b/mllib/src/main/scala/org/apache/spark/ml/feature/DCT.scala
index fe240557c0fa..d6e66d83aed6 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/feature/DCT.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/feature/DCT.scala
@@ -62,11 +62,23 @@ class DCT @Since("1.5.0") (@Since("1.5.0") override val
uid: String)
setDefault(inverse -> false)
- override protected def createTransformFunc: Vector => Vector = { vec =>
- val result = vec.toArray
- val jTransformer = new DoubleDCT_1D(result.length)
- if ($(inverse)) jTransformer.inverse(result, true) else
jTransformer.forward(result, true)
- Vectors.dense(result)
+ override protected def createTransformFunc: Vector => Vector = {
+ $(inverse) match {
+ case true =>
+ (vec: Vector) => {
+ val result = vec.toArray
+ val jTransformer = new DoubleDCT_1D(result.length)
+ jTransformer.inverse(result, true)
+ Vectors.dense(result)
+ }
+ case false =>
+ (vec: Vector) => {
+ val result = vec.toArray
+ val jTransformer = new DoubleDCT_1D(result.length)
+ jTransformer.forward(result, true)
+ Vectors.dense(result)
+ }
+ }
}
override protected def validateInputType(inputType: DataType): Unit = {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
