xuechendi opened a new pull request #32717: URL: https://github.com/apache/spark/pull/32717
Fixed: https://issues.apache.org/jira/browse/SPARK-35396 Signed-off-by: Chendi Xue <[email protected]> ### What changes were proposed in this pull request? This PR is used to do manual close for some objects may not be released by GC. For example some arrow allocated memory or other native objects. ### Why are the changes needed? Added a case match in InMemoryRelation 'clearCache' function, if one object is extends from AutoCloseable, then it will manually call its close function in case there is additional memory should be manually released. So one can implement CachedBatch extends from AutoCloseable to indidate this object require extra release. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? UT is added, org.apache.spark.sql.execution.columnar.RefCountedTestCachedBatchSerializerSuite -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
