This is an automated email from the ASF dual-hosted git repository.
sarutak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 67b62cb4081e [SPARK-60419][CORE] Fix file descriptor leak in
`FileSystemPersistenceEngine.deserializeFromFile` when compression is enabled
67b62cb4081e is described below
commit 67b62cb4081eda45933e76797a7730596dc50369
Author: Kousuke Saruta <[email protected]>
AuthorDate: Thu Oct 22 22:27:41 2026 +0900
[SPARK-60419][CORE] Fix file descriptor leak in
`FileSystemPersistenceEngine.deserializeFromFile` when compression is enabled
### What changes were proposed in this pull request?
Fix a file descriptor leak in
`FileSystemPersistenceEngine.deserializeFromFile` that occurs when a recovery
compression codec is configured (`spark.deploy.recoveryCompressionCodec`).
Before:
```scala
private def deserializeFromFile[T](file: File)(implicit m: ClassTag[T]): T
= {
var fileIn: InputStream = new FileInputStream(file)
codec.foreach { c => fileIn = c.compressedInputStream(new
FileInputStream(file)) }
var in: DeserializationStream = null
try {
in = serializer.newInstance().deserializeStream(fileIn)
in.readObject[T]()
} finally {
fileIn.close()
if (in != null) {
in.close()
}
}
}
```
When a codec is set, the `codec.foreach` line opens a **second**
`FileInputStream(file)` and reassigns `fileIn`. The first `FileInputStream`
opened on the line above then loses its only reference, so the `finally`
block's `fileIn.close()` closes only the second (wrapper) stream and the first
descriptor leaks.
The write-side sibling `serializeIntoFile` in the same file is already
correct: it wraps the existing stream (`fileOut =
c.compressedOutputStream(fileOut)`) rather than opening a new one. This PR
makes the read side symmetric with it.
After:
```scala
private def deserializeFromFile[T](file: File)(implicit m: ClassTag[T]): T
= {
var fileIn: InputStream = new FileInputStream(file)
var in: DeserializationStream = null
Utils.tryWithSafeFinally {
codec.foreach { c => fileIn = c.compressedInputStream(fileIn) }
in = serializer.newInstance().deserializeStream(fileIn)
in.readObject[T]()
} {
if (in != null) {
in.close()
}
fileIn.close()
}
}
```
Three points:
1. `c.compressedInputStream(fileIn)` wraps the existing stream instead of
opening a new one, so only one `FileInputStream` is ever opened and it is
always the one the `finally` block closes.
2. The codec wrapping is moved **inside** the `tryWithSafeFinally` body.
Some codecs read from the stream in their constructor (for example
`SnappyInputStream` reads and validates the header eagerly), so wrapping can
throw on a truncated or partially written recovery file. By doing it inside the
body, the `finally` block still runs and closes the raw `fileIn` when the wrap
itself throws (previously both `FileInputStream`s leaked on that path). The
`deserializeStream`-failure path is u [...]
3. The plain `try/finally` is switched to `Utils.tryWithSafeFinally` and
the two closes are reordered to close the outer `in` before the inner `fileIn`,
matching `serializeIntoFile` exactly. With `tryWithSafeFinally`, an exception
thrown while closing no longer masks the real deserialization error (it is
attached via `addSuppressed` and logged instead).
Scope note: the write side (`serializeIntoFile`) is already correct for the
descriptor leak and is left unchanged; this PR targets only the read path where
the leak occurs.
### Why are the changes needed?
A standalone Master configured with `spark.deploy.recoveryCompressionCodec`
and `FILESYSTEM` recovery leaks one file descriptor per persisted
`app_`/`driver_`/`worker_` file when it reads the recovery directory at startup
(once per Master process, from the `ElectedLeader` handler), until a GC lets
the JDK `FileInputStream` cleaner close it.
The leak is silent: the existing round-trip tests do not detect it, and the
orphaned descriptor is only reclaimed later by the JDK cleaner on GC rather
than by an explicit `close()`. It is a one-time spike of up to one extra
descriptor per recovery file at startup; it does not accumulate across reads,
since a Master process reads the recovery directory only once. It is still an
unambiguous resource-handling bug, and the read side is asymmetric with the
correct write side.
### Does this PR introduce _any_ user-facing change?
No. This is an internal resource-handling fix. Round-trip (de)serialization
behavior is unchanged for every codec: the bytes handed to the codec are
identical (a fresh stream at offset 0 in both the old and new code), so there
is no change in what is read or written.
### How was this patch tested?
Existing tests in `PersistenceEngineSuite` pass, including `SPARK-46216:
FileSystemPersistenceEngine with compression`, which round-trips through every
codec in `ALL_COMPRESSION_CODECS`, and `RecoverySuite`'s compression coverage.
No new regression test is added, intentionally. The leaked stream is the
first `FileInputStream`, which is never handed to the codec, so a test-only
`CompressionCodec` cannot observe it. The only way to observe the leak directly
is to count process-level file descriptors (for example via
`UnixOperatingSystemMXBean.getOpenFileDescriptorCount`), which is
platform-dependent and flaky in a shared test JVM (descriptors are reclaimed by
the JDK cleaner on GC and the count is JVM-wide); Spar [...]
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Kiro CLI (auto model selection)
Closes #59670 from sarutak/SPARK-fspe-fileinput-leak.
Authored-by: Kousuke Saruta <[email protected]>
Signed-off-by: Kousuke Saruta <[email protected]>
---
.../apache/spark/deploy/master/FileSystemPersistenceEngine.scala | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git
a/core/src/main/scala/org/apache/spark/deploy/master/FileSystemPersistenceEngine.scala
b/core/src/main/scala/org/apache/spark/deploy/master/FileSystemPersistenceEngine.scala
index 6df3c188def7..534a6b89ad1f 100644
---
a/core/src/main/scala/org/apache/spark/deploy/master/FileSystemPersistenceEngine.scala
+++
b/core/src/main/scala/org/apache/spark/deploy/master/FileSystemPersistenceEngine.scala
@@ -89,16 +89,16 @@ private[master] class FileSystemPersistenceEngine(
// still be shared across hosts (e.g. an NFS-mounted recovery directory).
private def deserializeFromFile[T](file: File)(implicit m: ClassTag[T]): T =
{
var fileIn: InputStream = new FileInputStream(file)
- codec.foreach { c => fileIn = c.compressedInputStream(new
FileInputStream(file)) }
var in: DeserializationStream = null
- try {
+ Utils.tryWithSafeFinally {
+ codec.foreach { c => fileIn = c.compressedInputStream(fileIn) }
in = serializer.newInstance().deserializeStream(fileIn)
in.readObject[T]()
- } finally {
- fileIn.close()
+ } {
if (in != null) {
in.close()
}
+ fileIn.close()
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]