Chendi.Xue created SPARK-35396:
----------------------------------
Summary: Support to manual close entries in MemoryStore and
InMemoryRelation
Key: SPARK-35396
URL: https://issues.apache.org/jira/browse/SPARK-35396
Project: Spark
Issue Type: New Feature
Components: Spark Core, SQL
Affects Versions: 3.1.1
Reporter: Chendi.Xue
Current MemoryStore uses a LinkedHashMap[BlockId, MemoryEntry[_]] to store all
OnHeap or OffHeap entries.
And when memoryStore.remove(blockId) is called, codes will simply remove one
entry from LinkedHashMap and leverage Java GC to do release work.
We are proposing a add-on, if this object is extends from AutoCloseable, then
we can call this object's close() directly in MemoryStore.clear() and
MemoryStore.remove() function.
In our case, we are implementing a user-defined off-heap-hashRelation for BHJ,
and we verified that by adding this manual close, we can make sure our defined
off-heap-hashRelation can be released when evict is called.
At same, time, we also want to add this logic to InMemoryRelation to manual
close a user-defined CachedBatch
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
