Github user sheperdh commented on the issue:
https://github.com/apache/spark/pull/16909
I have one question regarding this change.
We are doing some tests using spark-sql.
In our SQL, we have used "Full Outer Join", we found it's quite easy to hit
OOM when there is large dataset.
After analyzed the heap dump, we found it's because for "Full Outer Join",
we haven't used the ExternalAppendOnlyUnsafeRowArray.
I am not sure whether there is any special reason why Spark-SQL doesn't
optimize "Full Outer Join".
@tejasapatil
Can you let me know why you don't want to support "Full Outer Join"?
thanks,
Sheperd
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
