Github user shahidki31 commented on the issue:
https://github.com/apache/spark/pull/22645
Test step to reproduce OOM without the PR.
1) bin/spark-shell --conf spark.sql.ui.retainedExecutions=50000
for (i <- 0 until 50000) {
val df = Seq(
(1, 1),
(2, 2)
).toDF()
df.collect()
Without the PR:
After fix:
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
