Github user zhzhan commented on the issue:
https://github.com/apache/spark/pull/16909
@hvanhovell @davies Correct me if I am wrong. My understanding is that
following code will go though all matching rows on the right side, and put them
into the BufferedRowIterator. If there is OOM caused by ArrayList matches in
SortMergeJoinExec, the memory usage will be doubled in currentRows in
BufferedRowIterator (assuming the left and right have the same size).
There are two way to solve it. One is to consume one row at a time, and the
other one is make BufferedRowIterator spoilable (which should be much easier
based on this PR).
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala#L556
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
