[ https://issues.apache.org/jira/browse/SPARK-35396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean R. Owen resolved SPARK-35396. ---------------------------------- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32534 [https://github.com/apache/spark/pull/32534] > Support to manual close/release entries in MemoryStore and InMemoryRelation > instead of replying on GC > ----------------------------------------------------------------------------------------------------- > > Key: SPARK-35396 > URL: https://issues.apache.org/jira/browse/SPARK-35396 > Project: Spark > Issue Type: New Feature > Components: Spark Core, SQL > Affects Versions: 3.0.3, 3.1.2, 3.2.0 > Reporter: Chendi.Xue > Assignee: Apache Spark > Priority: Major > Fix For: 3.2.0 > > > This PR is proposing a add-on to support to manual close entries in > MemoryStore and InMemoryRelation > h3. What changes were proposed in this pull request? > Currently: > MemoryStore uses a LinkedHashMap[BlockId, MemoryEntry[_]] to store all OnHeap > or OffHeap entries. > And when memoryStore.remove(blockId) is called, codes will simply remove one > entry from LinkedHashMap and leverage Java GC to do release work. > This PR: > We are proposing a add-on to manually close any object stored in MemoryStore > and InMemoryRelation if this object is extended from AutoCloseable. > Veifiication: > In our own use case, we implemented a user-defined off-heap-hashRelation for > BHJ, and we verified that by adding this manual close, we can make sure our > defined off-heap-hashRelation can be released when evict is called. > Also, we implemented user-defined cachedBatch and will be release when > InMemoryRelation.clearCache() is called by this PR > h3. Why are the changes needed? > This changes can help to clean some off-heap user-defined object may be > cached in InMemoryRelation or MemoryStore > h3. Does this PR introduce _any_ user-facing change? > NO > h3. How was this patch tested? > WIP > Signed-off-by: Chendi Xue [chendi....@intel.com|mailto:chendi....@intel.com] -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org