This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new decdc16ee718 [SPARK-57958][SQL][TEST] Close the ORC Reader in
OrcSourceSuite to fix leaked file streams
decdc16ee718 is described below
commit decdc16ee718fdcf073ee0c1406ebea18f3d57dd
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Jul 6 18:53:18 2026 +0900
[SPARK-57958][SQL][TEST] Close the ORC Reader in OrcSourceSuite to fix
leaked file streams
### What changes were proposed in this pull request?
In `OrcSuite.testBloomFilterCreation` and
`testSelectiveDictionaryEncoding`, also close the ORC `Reader` (returned by
`OrcFile.createReader`) in the `finally` block — previously only the derived
`RecordReader` was closed.
### Why are the changes needed?
Both helpers do:
```scala
val reader = OrcFile.createReader(orcFilePath, readerOptions)
var recordReader: RecordReaderImpl = null
try {
recordReader = reader.rows.asInstanceOf[RecordReaderImpl]
...
} finally {
if (recordReader != null) recordReader.close() // Reader itself is
never closed
}
```
The `Reader` holds the file stream opened in
`org.apache.orc.impl.ReaderImpl.extractFileTail`, so it leaks.
`SharedSparkSessionBase.afterEach` asserts no open streams
(`DebugFilesystem.assertNoOpenStreams`) and aborts the whole suite when the
leaked stream is not closed within its wait window:
```
org.apache.spark.sql.execution.datasources.orc.OrcSourceV1Suite *** ABORTED
***
... There are 1 possibly leaked file streams.
(SharedSparkSession.scala:197)
```
The leaked-connection stack (logged by `DebugFilesystem`) points at
`org.apache.orc.impl.ReaderImpl.extractFileTail` / `OrcFile.createReader`. This
flakes `OrcSourceV1Suite`/`OrcSourceV2Suite` (observed repeatedly on
`build_java21` for branch-4.0). Closing the `Reader` fixes the leak.
### Does this PR introduce _any_ user-facing change?
No, test-only.
### How was this patch tested?
Ran `OrcSourceV1Suite` and `OrcSourceV2Suite` on GitHub Actions.
- **Before (failing on `apache/spark` `build_java21` / branch-4.0):**
https://github.com/apache/spark/actions/runs/28766468775/job/85293800991 —
`OrcSourceV1Suite *** ABORTED ***`, `There are 1 possibly leaked file streams.
(SharedSparkSession.scala:197)`.
- **After (passing with this change):**
https://github.com/HyukjinKwon/spark-agent6/actions/runs/28781272553/job/85337180343
— `OrcSourceV1Suite` + `OrcSourceV2Suite`: `Tests: succeeded 72, failed 0`,
`All tests passed.` (no leaked-stream abort).
### Was this patch authored or co-authored using generative AI tooling?
Yes.
This pull request and its description were written by Isaac.
Closes #57036 from HyukjinKwon/ci-fix/agent5-orc-reader-leak.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
(cherry picked from commit 9633f932aec024e067238b8df9ef16491bd4df4d)
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.../org/apache/spark/sql/execution/datasources/orc/OrcSourceSuite.scala | 2 ++
1 file changed, 2 insertions(+)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcSourceSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcSourceSuite.scala
index 040999476ece..0b039e2fdc78 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcSourceSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcSourceSuite.scala
@@ -121,6 +121,7 @@ abstract class OrcSuite
if (recordReader != null) {
recordReader.close()
}
+ reader.close()
}
}
}
@@ -191,6 +192,7 @@ abstract class OrcSuite
if (recordReader != null) {
recordReader.close()
}
+ reader.close()
}
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
