Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21502#discussion_r194101781
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/joins/BroadcastJoinSuite.scala
---
@@ -153,6 +154,23 @@ class BroadcastJoinSuite extends QueryTest with
SQLTestUtils {
}
test("SPARK-22575: remove allocated blocks when they are not needed
anymore") {
+ val blockManager = sparkContext.env.blockManager
+ def broadcastedBlockIds: Seq[BlockId] = {
+ blockManager.getMatchingBlockIds(blockId => {
+ blockId.isBroadcast &&
blockManager.getStatus(blockId).get.storageLevel.deserialized
+ }).distinct
+ }
+ def isHashedRelationPresent(blockIds: Seq[BlockId]): Boolean = {
+ val blockValues = blockIds.flatMap { id =>
+ val block = blockManager.getSingle[Any](id)
+ if (block.isDefined) {
--- End diff --
no, the reason of the last failure was that the tests which run before this
test case were not cleaning up their broadcasted variables. So the UT was
failing because there were broadcasted items present in the block manager, but
not because of this test, because of the leftovers of previous tests. The
reason why the test was passing for you adding a sleep was that in that time,
the previous broadcasted items went cleaned up by the apposite thread.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
