Github user JeetKunDoug commented on a diff in the pull request:
https://github.com/apache/spark/pull/21322#discussion_r188074445
--- Diff:
core/src/test/scala/org/apache/spark/storage/MemoryStoreSuite.scala ---
@@ -526,4 +526,84 @@ class MemoryStoreSuite
}
}
}
+
+ test("[SPARK-24225]: remove should close AutoCloseable object") {
+
+ val (store, _) = makeMemoryStore(12000)
+
+ val id = BroadcastBlockId(0)
+ val tracker = new CloseTracker()
+ store.putIteratorAsValues(id, Iterator(tracker), ClassTag.Any)
+ assert(store.contains(id))
+ store.remove(id)
+ assert(tracker.getClosed())
+ }
+
+ test("[SPARK-24225]: remove should close AutoCloseable objects even if
they throw exceptions") {
+
+ val (store, _) = makeMemoryStore(12000)
+
+ val id = BroadcastBlockId(0)
+ val tracker = new CloseTracker(true)
+ store.putIteratorAsValues(id, Iterator(tracker), ClassTag.Any)
+ assert(store.contains(id))
+ store.remove(id)
+ assert(tracker.getClosed())
+ }
+
+ test("[SPARK-24225]: clear should close AutoCloseable objects") {
+
+ val (store, _) = makeMemoryStore(12000)
+
+ val id = BroadcastBlockId(0)
+ val tracker = new CloseTracker
+ store.putIteratorAsValues(id, Iterator(tracker), ClassTag.Any)
+ assert(store.contains(id))
+ store.clear()
+ assert(tracker.getClosed())
+ }
+
+ test("[SPARK-24225]: clear should close all AutoCloseable objects put
together in an iterator") {
--- End diff --
So if I understand the API correctly, there is no way to remove a single
item that was put as part of a call to `putIterator` - because operations are
conducted by `blockId` you would only be able to remove the whole group of
entries, not a single part of an iterator.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
