[
https://issues.apache.org/jira/browse/SPARK-61248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61248:
-----------------------------------
Labels: pull-request-available (was: )
> Sort-merge ASOF join leaves spill file readers open for each left row
> ---------------------------------------------------------------------
>
> Key: SPARK-61248
> URL: https://issues.apache.org/jira/browse/SPARK-61248
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Luka Zdravic
> Priority: Minor
> Labels: pull-request-available
>
> The buffered scan of SortMergeAsOfJoinScanner (findLastBackward,
> findFirstForward and findNearest) makes a new iterator over the right-side
> group for each left row. Once the group has spilled, that iterator opens a
> reader on every spill file. The scan usually returns before the iterator ends
> and never closes it, so each reader and its read buffer stay open until the
> task ends.
>
> With L left rows in a group spilled to S files, a task can hold L * S open
> readers, which can fail the task with "Too many open files" or run it out of
> memory.
> Close the iterator before each early return.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
