This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new eacf31bf9a42 [SPARK-60467][CORE] Fix `FileSystemPersistenceEngine` to 
clean up failed writes and skip unreadable files
eacf31bf9a42 is described below

commit eacf31bf9a429d28e34ebe20ad7c7444c3d026bd
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Oct 22 15:23:50 2026 -0700

    [SPARK-60467][CORE] Fix `FileSystemPersistenceEngine` to clean up failed 
writes and skip unreadable files
    
    ### What changes were proposed in this pull request?
    
    This PR fixes two robustness issues in `FileSystemPersistenceEngine`, which 
backs the `FILESYSTEM` recovery mode of the standalone Master.
    
    1. `serializeIntoFile` keeps the raw `FileOutputStream` as `rawOut` and 
closes the serialization stream, the codec stream and `rawOut`, in this order, 
each guaranteed by a nested `Utils.tryWithSafeFinally`. The close order from 
SPARK-46205, where the serializer and the codec flush before the file closes, 
is unchanged. The codec stream is now created inside the `tryWithSafeFinally` 
body (as SPARK-60419 does on the read side), so the raw stream is closed even 
when the codec stream const [...]
    2. `serializeIntoFile` deletes the file and rethrows when writing fails 
after it created the file. It never deletes an existing file: `File already 
exists` and `Could not create file` are thrown before the file is created.
    3. `read` skips a file that cannot be deserialized instead of failing as a 
whole. It logs a warning with the path and the exception, and renames the file 
to `corrupted_<name>`, which no longer matches the `app_`, `driver_` or 
`worker_` prefix and is kept for inspection.
    
    Items 2 and 3 together are the smallest design that unblocks recovery in 
all cases:
    - Deleting on write failures alone does not cover a crash or a power loss 
in the middle of a write, or files that released versions already left behind.
    - Skipping on read alone keeps a failed write's file under its real name 
until the next start, which then warns about it and renames it.
    - Writing to a temporary file, then `fsync` and `ATOMIC_MOVE`, would add an 
`fsync` to every app, driver and worker registration on the Master's RPC 
thread, which is slow on an NFS-mounted recovery directory. It also needs 
cleanup of leftover temporary files, `ATOMIC_MOVE` support from the file 
system, and a directory `fsync` that Java cannot do portably, and it still does 
not cover files that are already broken. Durability is out of the scope of this 
PR.
    
    Unlike `ZooKeeperPersistenceEngine`, which deletes unreadable entries, 
`read` renames them. This keeps the data when the cause is not a broken file, 
e.g. an incompatible class, a changed `spark.deploy.recoveryCompressionCodec`, 
or a transient I/O error.
    
    SPARK-60419 (https://github.com/apache/spark/pull/59670) fixed the double 
open in `deserializeFromFile`. This PR keeps that method as is, and fixes the 
write side and the per-file handling in `read`.
    
    ### Why are the changes needed?
    
    **Stream leaks on write failures.** When the last write fails during close, 
e.g. with `ENOSPC` or `EDQUOT`, `out.close()` throws. 
`ObjectOutputStream.close()` calls `flush()` first, which pushes the buffered 
data through the codec, so it never reaches the codec stream's `close()`. The 
`finally` block then skips `fileOut.close()`:
    - The raw `FileOutputStream` stays open until GC, with or without a codec.
    - With zstd, `ZstdOutputStreamNoFinalizer` is never closed, so its native 
`CStream` is never freed. The class has no finalizer as a fallback.
    
    Reaching `fileOut.close()` is not enough. When their final write fails, the 
`close()` methods of `LZ4BlockOutputStream`, `ZstdOutputStreamNoFinalizer` and 
the non-parallel `LZFOutputStream` throw before closing the stream they wrap. 
So does `SnappyOutputStream` in snappy-java 1.1.10.8 and older, which 
`branch-4.x`, `branch-4.1` and `branch-4.0` use; snappy-java 1.1.10.9 fixed it 
in https://github.com/xerial/snappy-java/pull/744. Nothing else referenced the 
raw `FileOutputStream`. The  [...]
    
    **Partially written files block recovery.** `serializeIntoFile` creates the 
file before writing it, never deletes it on failure, and does not `fsync`. 
`read` fails as a whole when any file cannot be deserialized. So an empty or 
truncated `app_`, `driver_` or `worker_` file, left by a failed write, a crash 
in the middle of a write, or a power loss, makes `readPersistedData` throw in 
the `ElectedLeader` handler on the next start. The RPC inbox only logs 
`Ignoring error`, the Master stay [...]
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, in the `FILESYSTEM` recovery mode, compared with the released versions:
    - A failed persist no longer leaves an empty or truncated file in the 
recovery directory, and no longer leaks its file descriptor.
    - An unreadable file in the recovery directory no longer keeps the Master 
in `STANDBY`. The Master logs a warning, renames the file to 
`corrupted_<name>`, and recovers the other apps, drivers and workers. This also 
applies when the recovery directory was not cleared after changing 
`spark.deploy.recoveryMode` or `spark.deploy.recoveryCompressionCodec`, which 
the documentation asks users to do.
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test cases, which fail without the main 
code change:
    - `FileSystemPersistenceEngine closes and deletes the file when persist 
fails` passes a test-only `CompressionCodec` through the engine's `codec` 
constructor parameter. Its output stream fails on `flush()` and `close()` 
without closing the stream it wraps, as the lz4, zstd and lzf streams do on a 
full disk, or it fails to be created. The test checks that the codec stream's 
`close()` is called, that the raw `FileOutputStream` is closed (`getFD.valid()` 
is `false`), and that no file is  [...]
    - `FileSystemPersistenceEngine skips and renames unreadable files` writes 
an empty file and a truncated file next to a valid one. `read` returns only the 
valid object, the two files are renamed to `corrupted_<name>`, and their names 
can be persisted again. Without the fix, `read` throws `java.io.EOFException`.
    - The existing `SPARK-46191` test case also checks that a persist failing 
with `File already exists` keeps the existing file.
    
    ```
    $ build/sbt 'core/testOnly *PersistenceEngineSuite *RecoverySuite'
    ...
    [info] Tests: succeeded 28, failed 0, canceled 0, ignored 0, pending 0
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #59709 from dongjoon-hyun/SPARK-60467.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../master/FileSystemPersistenceEngine.scala       | 59 ++++++++++++++++----
 .../deploy/master/PersistenceEngineSuite.scala     | 64 ++++++++++++++++++++++
 2 files changed, 112 insertions(+), 11 deletions(-)

diff --git 
a/core/src/main/scala/org/apache/spark/deploy/master/FileSystemPersistenceEngine.scala
 
b/core/src/main/scala/org/apache/spark/deploy/master/FileSystemPersistenceEngine.scala
index 534a6b89ad1f..8f0369ae5b6b 100644
--- 
a/core/src/main/scala/org/apache/spark/deploy/master/FileSystemPersistenceEngine.scala
+++ 
b/core/src/main/scala/org/apache/spark/deploy/master/FileSystemPersistenceEngine.scala
@@ -21,6 +21,7 @@ import java.io._
 import java.nio.file.{FileAlreadyExistsException, Files, Paths}
 
 import scala.reflect.ClassTag
+import scala.util.control.NonFatal
 
 import org.apache.spark.internal.Logging
 import org.apache.spark.internal.LogKeys._
@@ -63,24 +64,60 @@ private[master] class FileSystemPersistenceEngine(
 
   override def read[T: ClassTag](prefix: String): Seq[T] = {
     val files = new File(dir).listFiles().filter(_.getName.startsWith(prefix))
-    files.map(deserializeFromFile[T]).toImmutableArraySeq
+    files.flatMap { file =>
+      try {
+        Some(deserializeFromFile[T](file))
+      } catch {
+        case NonFatal(e) =>
+          // Skip an unreadable file, e.g. one left truncated by a crash, so 
that it cannot block
+          // recovery. Rename it, instead of deleting it, so that it is kept 
for inspection but no
+          // longer matches the app_, driver_ or worker_ prefix.
+          val corrupted = new File(dir, "corrupted_" + file.getName)
+          logWarning(log"Exception while reading persisted file ${MDC(PATH, 
file.getPath())}, " +
+            log"renaming it to ${MDC(NEW_PATH, corrupted.getPath())}", e)
+          if (!file.renameTo(corrupted)) {
+            logWarning(log"Error renaming ${MDC(PATH, file.getPath())}")
+          }
+          None
+      }
+    }.toImmutableArraySeq
   }
 
   private def serializeIntoFile(file: File, value: AnyRef): Unit = {
     if (file.exists()) { throw new IllegalStateException("File already exists: 
" + file) }
     val created = file.createNewFile()
     if (!created) { throw new IllegalStateException("Could not create file: " 
+ file) }
-    var fileOut: OutputStream = new FileOutputStream(file)
-    codec.foreach { c => fileOut = c.compressedOutputStream(fileOut) }
-    var out: SerializationStream = null
-    Utils.tryWithSafeFinally {
-      out = serializer.newInstance().serializeStream(fileOut)
-      out.writeObject(value)
-    } {
-      if (out != null) {
-        out.close()
+    try {
+      val rawOut = new FileOutputStream(file)
+      var fileOut: OutputStream = rawOut
+      var out: SerializationStream = null
+      Utils.tryWithSafeFinally {
+        codec.foreach { c => fileOut = c.compressedOutputStream(fileOut) }
+        out = serializer.newInstance().serializeStream(fileOut)
+        out.writeObject(value)
+      } {
+        // Close the serializer and the codec before the file so that they 
flush first. Close
+        // each one even if the previous close fails, e.g. on a full disk, 
because some codec
+        // streams do not close the stream they wrap when their final write 
fails.
+        Utils.tryWithSafeFinally {
+          Utils.tryWithSafeFinally {
+            if (out != null) {
+              out.close()
+            }
+          } {
+            fileOut.close()
+          }
+        } {
+          rawOut.close()
+        }
       }
-      fileOut.close()
+    } catch {
+      case e: Throwable =>
+        // Remove the file created above so that a partially written file 
cannot block recovery.
+        if (!file.delete()) {
+          logWarning(log"Error deleting ${MDC(PATH, file.getPath())}")
+        }
+        throw e
     }
   }
 
diff --git 
a/core/src/test/scala/org/apache/spark/deploy/master/PersistenceEngineSuite.scala
 
b/core/src/test/scala/org/apache/spark/deploy/master/PersistenceEngineSuite.scala
index 416af7457432..087478c89fe9 100644
--- 
a/core/src/test/scala/org/apache/spark/deploy/master/PersistenceEngineSuite.scala
+++ 
b/core/src/test/scala/org/apache/spark/deploy/master/PersistenceEngineSuite.scala
@@ -18,6 +18,7 @@
 
 package org.apache.spark.deploy.master
 
+import java.io.{File, FileOutputStream, FilterOutputStream, InputStream, 
IOException, OutputStream}
 import java.net.ServerSocket
 import java.nio.file.{Files, Paths}
 import java.util.Date
@@ -64,6 +65,8 @@ class PersistenceEngineSuite extends SparkFunSuite {
         engine.persist("test_1", "test_1_value")
       }.getMessage
       assert(m.contains("File already exists"))
+      // The failed persist does not delete the existing file.
+      assert(engine.read[String]("test_1") === Seq("test_1_value"))
     }
   }
 
@@ -118,6 +121,42 @@ class PersistenceEngineSuite extends SparkFunSuite {
     }
   }
 
+  test("SPARK-60467: FileSystemPersistenceEngine closes and deletes the file 
when persist fails") {
+    Seq(false, true).foreach { failOnCreate =>
+      withTempDir { dir =>
+        val codec = new FailingCompressionCodec(failOnCreate)
+        val engine = new FileSystemPersistenceEngine(
+          dir.getAbsolutePath, new JavaSerializer(new SparkConf()), 
Some(codec))
+        intercept[IOException] {
+          engine.persist("test_1", "test_1_value")
+        }
+        // The codec stream is closed if it was created, and the file stream 
is closed although
+        // the codec stream does not close it.
+        assert(codec.closeCalled === !failOnCreate)
+        assert(!codec.fileOut.getFD.valid())
+        // No partially written file is left behind.
+        assert(dir.list().isEmpty)
+      }
+    }
+  }
+
+  test("SPARK-60467: FileSystemPersistenceEngine skips and renames unreadable 
files") {
+    withTempDir { dir =>
+      val engine = new FileSystemPersistenceEngine(
+        dir.getAbsolutePath, new JavaSerializer(new SparkConf()))
+      engine.persist("test_1", "test_1_value")
+      // An empty file and a truncated file, as a crash in the middle of a 
persist can leave.
+      val bytes = Files.readAllBytes(new File(dir, "test_1").toPath)
+      Files.write(new File(dir, "test_2").toPath, Array.emptyByteArray)
+      Files.write(new File(dir, "test_3").toPath, bytes.take(bytes.length / 2))
+      assert(engine.read[String]("test_") === Seq("test_1_value"))
+      assert(dir.list().toSet === Set("test_1", "corrupted_test_2", 
"corrupted_test_3"))
+      // The renamed files neither block reads nor persisting the same names 
again.
+      engine.persist("test_2", "test_2_value")
+      assert(engine.read[String]("test_").toSet === Set("test_1_value", 
"test_2_value"))
+    }
+  }
+
   test("ZooKeeperPersistenceEngine") {
     val conf = new SparkConf()
     // TestingServer logs the port conflict exception rather than throwing an 
exception.
@@ -272,4 +311,29 @@ class PersistenceEngineSuite extends SparkFunSuite {
       (null, trialPort)
     }, conf)._2
   }
+
+  /**
+   * A codec that fails to create its output stream, or whose output stream 
fails to flush and
+   * close, as on a full disk, without closing the file stream it wraps.
+   */
+  private class FailingCompressionCodec(failOnCreate: Boolean) extends 
CompressionCodec {
+    var fileOut: FileOutputStream = _
+    var closeCalled = false
+
+    override def compressedOutputStream(s: OutputStream): OutputStream = {
+      fileOut = s.asInstanceOf[FileOutputStream]
+      if (failOnCreate) {
+        throw new IOException("Cannot create the compressed stream")
+      }
+      new FilterOutputStream(s) {
+        override def flush(): Unit = throw new IOException("No space left on 
device")
+        override def close(): Unit = {
+          closeCalled = true
+          throw new IOException("No space left on device")
+        }
+      }
+    }
+
+    override def compressedInputStream(s: InputStream): InputStream = s
+  }
 }


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to