[ 
https://issues.apache.org/jira/browse/SPARK-61251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61251:
-----------------------------------
    Labels: pull-request-available  (was: )

> Sort-merge join and window frames leave spill file readers open when they 
> stop reading a spilled buffer early
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61251
>                 URL: https://issues.apache.org/jira/browse/SPARK-61251
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.4.0
>            Reporter: Luka Zdravic
>            Priority: Minor
>              Labels: pull-request-available
>
>   SPARK-61248 fixed this for the sort-merge ASOF join. Other code that reads 
> an ExternalAppendOnlyUnsafeRowArray has the same problem: it makes a new 
> iterator over the buffer for each row and stops before the end. Once the 
> buffer has spilled, that iterator keeps a reader open on every spill file it 
> has not read, until the task ends. With L rows and S spill files, a task 
> keeps up to L * S files open and can fail with "Too many open files".
>   Places found by reading the code:
>   - Sort-merge LeftSemi, LeftAnti and ExistenceJoin with a residual condition 
> stop at the first passing buffered row, in SortMergeJoinEvaluatorFactory and 
> in the generated codegenSemi, codegenAnti and codegenExistence loops.
>   - With spark.sql.join.splitStreamedSideJoinCondition on, the generated 
> sort-merge join makes the iterator before the streamed-side guard, and the 
> guard skips it unread.
>   - UnboundedFollowingWindowFunctionFrame.write reads one row and drops the 
> iterator when the frame's lower bound does not move, for example a RANGE 
> frame over duplicate order keys.
>   - WindowSegmentTree.buildBlockLevels (spark.sql.window.segmentTree.enabled) 
> reads one block and drops the iterator.
>   Example: SELECT * FROM l WHERE EXISTS (SELECT 1 FROM r WHERE r.k = l.k AND 
> r.v > l.v), where the buffered rows of a key spill.
>   Fix: close the iterator when the code stops reading it. In generated Java, 
> check instanceof
>   java.io.Closeable. In the generated sort-merge join, make the iterator 
> after the streamed-side guard.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to