(kyuubi) branch branch-1.9 updated: [KYUUBI #6400] Fix memory leak when using saveToFile

[feiwang]

This is an automated email from the ASF dual-hosted git repository.

feiwang pushed a commit to branch branch-1.9
in repository https://gitbox.apache.org/repos/asf/kyuubi.git


The following commit(s) were added to refs/heads/branch-1.9 by this push:
     new 6883552a8 [KYUUBI #6400] Fix memory leak when using saveToFile
6883552a8 is described below

commit 6883552a8142c06b1781c5e3d862386f949dac15
Author: Wang, Fei <fwan...@ebay.com>
AuthorDate: Mon May 20 19:30:39 2024 -0700

    [KYUUBI #6400] Fix memory leak when using saveToFile
    
    # :mag: Description
    ## Issue References 🔗
    
    Fix memory leak when using saveToFile mode.
    
    FYI: 
https://stackoverflow.com/questions/45649044/scala-stream-iterate-and-memory-management
    
    Stream is IterableAgain, which means, that it will keep all the elements 
you iterate through in case you want to see them again.
    
    ## Describe Your Solution 🔧
    
    Please include a summary of the change and which issue is fixed. Please 
also include relevant motivation and context. List any dependencies that are 
required for this change.
    
    ## Types of changes :bookmark:
    
    - [ ] Bugfix (non-breaking change which fixes an issue)
    - [ ] New feature (non-breaking change which adds functionality)
    - [ ] Breaking change (fix or feature that would cause existing 
functionality to change)
    
    ## Test Plan 🧪
    
    #### Behavior Without This Pull Request :coffin:
    
    #### Behavior With This Pull Request :tada:
    
    #### Related Unit Tests
    
    ---
    
    # Checklist 📝
    
    - [x] This patch was not authored or co-authored using [Generative 
Tooling](https://www.apache.org/legal/generative-tooling.html)
    
    **Be nice. Be informative.**
    
    Closes #6400 from turboFei/memory_leak.
    
    Closes #6400
    
    cdea358d6 [Wang, Fei] fix memory leak
    
    Authored-by: Wang, Fei <fwan...@ebay.com>
    Signed-off-by: Wang, Fei <fwan...@ebay.com>
    (cherry picked from commit 97aa4f7e1fd758adbe1314217ef51dd6a23cc9fd)
    Signed-off-by: Wang, Fei <fwan...@ebay.com>
---
 .../org/apache/kyuubi/engine/spark/operation/FetchOrcStatement.scala  | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git 
a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/FetchOrcStatement.scala
 
b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/FetchOrcStatement.scala
index 4673588c9..c19017077 100644
--- 
a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/FetchOrcStatement.scala
+++ 
b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/FetchOrcStatement.scala
@@ -67,7 +67,9 @@ class FetchOrcStatement(spark: SparkSession) {
     val iterRow = orcIter.map(value =>
       unsafeProjection(deserializer.deserialize(value)))
       .map(value => toRowConverter(value))
-    new IterableFetchIterator[Row](iterRow.toIterable)
+    new IterableFetchIterator[Row](new Iterable[Row] {
+      override def iterator: Iterator[Row] = iterRow
+    })
   }
 
   def close(): Unit = {