xuechendi commented on a change in pull request #32534:
URL: https://github.com/apache/spark/pull/32534#discussion_r633423094
##########
File path: core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala
##########
@@ -387,6 +388,14 @@ private[spark] class MemoryStore(
def remove(blockId: BlockId): Boolean = memoryManager.synchronized {
val entry = entries.synchronized {
+ entries.get(blockId) match {
Review comment:
@srowen , thx, I am a little concerned if I call entries.remove firstly,
will the returned obj already be cleared and it may encounter some reference
issue or other unexpected cases? so I still remained using get here.
##########
File path: core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala
##########
@@ -387,6 +388,14 @@ private[spark] class MemoryStore(
def remove(blockId: BlockId): Boolean = memoryManager.synchronized {
val entry = entries.synchronized {
+ entries.get(blockId) match {
+ case e: DeserializedMemoryEntry[_] => e.value.foreach(v => v match {
+ case o: AutoCloseable =>
Review comment:
@srowen , I fixed the codes based on your last review, 1) directly using
foreach { instead of foreach(v => v { 2) I added an async call here, so I hope
close functions won't hold memoryManager lock for too long 3) Added a try catch
to skip NonFatal exceptions.
##########
File path: core/src/test/scala/org/apache/spark/storage/MemoryStoreSuite.scala
##########
@@ -546,4 +551,105 @@ class MemoryStoreSuite
}
}
}
+
+ test("put native objects to MemoryStore and remove") {
Review comment:
@srowen , I added two UnitTests to verify this PR, my initial purpose of
adding this Manual close is to release memory allocated by Apache-Arrow, so
below UT is based on Arrow, while using Arrow in spark-core UT requires an
extra dependencies in core's pom, I am not sure if it is ok to add dependencies
in pom? Or I'll try to write something else to make the point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
