Luka Zdravic created SPARK-61251:
------------------------------------
Summary: Sort-merge join and window frames leave spill file
readers open when they stop reading a spilled buffer early
Key: SPARK-61251
URL: https://issues.apache.org/jira/browse/SPARK-61251
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.4.0
Reporter: Luka Zdravic
SPARK-61248 fixed this for the sort-merge ASOF join. Other code that reads an
ExternalAppendOnlyUnsafeRowArray has the same problem: it makes a new iterator
over the buffer for each row and stops before the end. Once the buffer has
spilled, that iterator keeps a reader open on every spill file it has not read,
until the task ends. With L rows and S spill files, a task keeps up to L * S
files open and can fail with "Too many open files".
Places found by reading the code:
- Sort-merge LeftSemi, LeftAnti and ExistenceJoin with a residual condition
stop at the first passing buffered row, in SortMergeJoinEvaluatorFactory and in
the generated codegenSemi, codegenAnti and codegenExistence loops.
- With spark.sql.join.splitStreamedSideJoinCondition on, the generated
sort-merge join makes the iterator before the streamed-side guard, and the
guard skips it unread.
- UnboundedFollowingWindowFunctionFrame.write reads one row and drops the
iterator when the frame's lower bound does not move, for example a RANGE frame
over duplicate order keys.
- WindowSegmentTree.buildBlockLevels (spark.sql.window.segmentTree.enabled)
reads one block and drops the iterator.
Example: SELECT * FROM l WHERE EXISTS (SELECT 1 FROM r WHERE r.k = l.k AND
r.v > l.v), where the buffered rows of a key spill.
Fix: close the iterator when the code stops reading it. In generated Java,
check instanceof
java.io.Closeable. In the generated sort-merge join, make the iterator after
the streamed-side guard.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
