This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 844dc3429f63 [SPARK-58683][ML] Optimize NGram transform closure
844dc3429f63 is described below
commit 844dc3429f63560715dec4530c6ffead42fbdc1e
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Mon Aug 10 20:14:01 2026 +0800
[SPARK-58683][ML] Optimize NGram transform closure
### What changes were proposed in this pull request?
Snapshot the configured n-gram size while creating NGram's transform
function. The resulting function no longer captures the NGram transformer.
The serialized transform function decreases from 2,988 bytes to 1,277 bytes
(57.3%).
### Why are the changes needed?
Avoiding the transformer capture reduces the memory retained by the
transform closure, which is especially useful for Spark Connect server
workloads.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Compiled the edited NGram source and serialized its generated transform
function with Java serialization. The existing NGramSuite covers the unchanged
transform behavior.
The targeted SBT suite could not run locally because dependency resolution
for `at.yawk:lz4-java:1.11.2` was denied by the configured Maven mirror.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
Closes #57886 from zhengruifeng/SPARK-58683-ngram-transform-state.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit c0d4ddaddaf6888939fa8e33da17d41672e846ad)
Signed-off-by: Ruifeng Zheng <[email protected]>
---
mllib/src/main/scala/org/apache/spark/ml/feature/NGram.scala | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/NGram.scala
b/mllib/src/main/scala/org/apache/spark/ml/feature/NGram.scala
index d72fb6ecc76d..3ef7a5a88719 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/feature/NGram.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/feature/NGram.scala
@@ -61,7 +61,8 @@ class NGram @Since("1.5.0") (@Since("1.5.0") override val
uid: String)
setDefault(n -> 2)
override protected def createTransformFunc: Seq[String] => Seq[String] = {
- _.iterator.sliding($(n)).withPartial(false).map(_.mkString(" ")).toSeq
+ val localN = $(n)
+ _.iterator.sliding(localN).withPartial(false).map(_.mkString(" ")).toSeq
}
override protected def validateInputType(inputType: DataType): Unit = {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
