SongYadong opened a new pull request #23585: [SPARK-26662][CORE] Dispose off-heap memory when MemoryStore is cleared URL: https://github.com/apache/spark/pull/23585 ## What changes were proposed in this pull request? If buffer in `MemoryStore`'s `SerializedMemoryEntry` is allocated off-heap, current `clear()` action doesn't dispose them. These off-heap memory, though, will be freed when process ends, but for application that need to stop and restart `SparkContext` serveral times, this may cause some kind of off-heap memory leak. This PR try to dispose off-heap memory by itself when clearing `MemoryStore`. ## How was this patch tested? Tested manually by running code written myself, while observing system resources monitor to verify off-heap memory allocation and free behavior. I have tried to add a unit test, but it's hard to find a way counting off-heap memory allocated by Unsafe API.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
