Github user liutang123 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21772#discussion_r204613880
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala
---
@@ -278,6 +278,39 @@ class HashedRelationSuite extends SparkFunSuite with
SharedSQLContext {
map.free()
}
+ test("SPARK-24809: Serializing LongHashedRelation in executor may result
in data error") {
--- End diff --
I think this UT can cover the case I had met.
End-to-end test is too hard to structure because this case just occurs when
executor's memory is not enough to hold the block and the broadcast cache is
removed by the garbage collector.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
