Luka Zdravic created SPARK-61248:
------------------------------------
Summary: Sort-merge ASOF join leaves spill file readers open for
each left row
Key: SPARK-61248
URL: https://issues.apache.org/jira/browse/SPARK-61248
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.4.0
Reporter: Luka Zdravic
The buffered scan of SortMergeAsOfJoinScanner (findLastBackward,
findFirstForward and findNearest) makes a new iterator over the right-side
group for each left row. Once the group has spilled, that iterator opens a
reader on every spill file. The scan usually returns before the iterator ends
and never closes it, so each reader and its read buffer stay open until the
task ends.
With L left rows in a group spilled to S files, a task can hold L * S open
readers, which can fail the task with "Too many open files" or run it out of
memory.
Close the iterator before each early return.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
