HyukjinKwon opened a new pull request, #57062: URL: https://github.com/apache/spark/pull/57062
> **[DO-NOT-MERGE]** — draft opened for CI validation of a branch-4.0 flake fix. Not for merge. ### What changes were proposed in this pull request? `OrcUtils.readSchema` previously read the ORC footer via `Utils.tryWithResource(OrcFile.createReader(...))`. This PR holds the `Reader` in an explicit variable and closes it in a `finally` block that **clears the thread's interrupt status around `Reader.close()`** (restoring it afterwards) and logs-and-swallows a non-fatal close failure, so the underlying file stream is always released. ### Why are the changes needed? `readSchema` runs on the `readingOrcSchemas` `parmap` worker threads (and Spark task threads) during parallel / merged ORC schema inference (`OrcUtils.readOrcSchemasInParallel` → `SchemaMergeUtils.mergeSchemasInParallel`). When a sibling file fails — e.g. the `SPARK-11412 read and merge orc schemas in parallel` test deliberately mixes a corrupt (JSON) file with `ignoreCorruptFiles=false` and asserts the Spark job aborts — the enclosing job is cancelled and a worker that has already opened a `Reader` can be interrupted before it is closed. The `Reader` holds an `FSDataInputStream` opened in `org.apache.orc.impl.ReaderImpl.extractFileTail`; because `Reader.close()` performs file-system I/O, a pending interrupt can prevent the stream from actually being released. In test builds the leaked stream is caught by `SharedSparkSessionBase.afterEach` → `DebugFilesystem.assertNoOpenStreams`, which retries for 10s then aborts the **whole suite** with `There are 1 possibly leaked file streams`. Because the leak is registered asynchronously, it surfaces nondeterministically in a *later* ORC suite's `afterEach`, so `OrcSourceV1Suite` / `OrcSourceV2Suite` abort intermittently on `build_java21` for `branch-4.0`: - https://github.com/apache/spark/actions/runs/28766468775 — `OrcSourceV1Suite *** ABORTED ***` - https://github.com/apache/spark/actions/runs/28731799013 — `OrcSourceV2Suite *** ABORTED ***` In both runs the leaked-connection stack logged by `DebugFilesystem` points exactly at: ``` DebugFilesystem.open → org.apache.orc.impl.ReaderImpl.extractFileTail → OrcFile.createReader → OrcUtils.readSchema(OrcUtils.scala:80) → OrcUtils.readOrcSchemasInParallel → ThreadUtils.parmap ``` [SPARK-57958](https://issues.apache.org/jira/browse/SPARK-57958) fixed a *sibling* reader leak in the `OrcSuite` test helpers (`testBloomFilterCreation` / `testSelectiveDictionaryEncoding`); those helpers do **not** appear as leak sites in the failing runs. This PR addresses the remaining production leak site those runs actually hit. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing `OrcSourceV1Suite` / `OrcSourceV2Suite`. The flake is a rare job-cancellation race that does not reproduce deterministically (verified: it did not surface across a standalone ORC repro, a 300× in-JVM loop of the exact merge-abort scenario, or 10× dedicated-JVM suite runs on GitHub Actions). This change makes the reader/stream release robust to interruption at the leak site the failing CI runs identified. This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
