This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new c0d4ddaddaf6 [SPARK-58683][ML] Optimize NGram transform closure
c0d4ddaddaf6 is described below
commit c0d4ddaddaf6888939fa8e33da17d41672e846ad
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Mon Aug 10 20:14:01 2026 +0800
[SPARK-58683][ML] Optimize NGram transform closure
### What changes were proposed in this pull request?
Snapshot the configured n-gram size while creating NGram's transform
function. The resulting function no longer captures the NGram transformer.
The serialized transform function decreases from 2,988 bytes to 1,277 bytes
(57.3%).
### Why are the changes needed?
Avoiding the transformer capture reduces the memory retained by the
transform closure, which is especially useful for Spark Connect server
workloads.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Compiled the edited NGram source and serialized its generated transform
function with Java serialization. The existing NGramSuite covers the unchanged
transform behavior.
The targeted SBT suite could not run locally because dependency resolution
for `at.yawk:lz4-java:1.11.2` was denied by the configured Maven mirror.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
Closes #57886 from zhengruifeng/SPARK-58683-ngram-transform-state.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
---
mllib/src/main/scala/org/apache/spark/ml/feature/NGram.scala | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/NGram.scala
b/mllib/src/main/scala/org/apache/spark/ml/feature/NGram.scala
index d72fb6ecc76d..3ef7a5a88719 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/feature/NGram.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/feature/NGram.scala
@@ -61,7 +61,8 @@ class NGram @Since("1.5.0") (@Since("1.5.0") override val
uid: String)
setDefault(n -> 2)
override protected def createTransformFunc: Seq[String] => Seq[String] = {
- _.iterator.sliding($(n)).withPartial(false).map(_.mkString(" ")).toSeq
+ val localN = $(n)
+ _.iterator.sliding(localN).withPartial(false).map(_.mkString(" ")).toSeq
}
override protected def validateInputType(inputType: DataType): Unit = {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
