This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new 0c0f92d8eabb [SPARK-60467][CORE] Fix `FileSystemPersistenceEngine` to
clean up failed writes and skip unreadable files
0c0f92d8eabb is described below
commit 0c0f92d8eabb164060e47647e791be44b22ccf5d
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Oct 22 15:23:50 2026 -0700
[SPARK-60467][CORE] Fix `FileSystemPersistenceEngine` to clean up failed
writes and skip unreadable files
### What changes were proposed in this pull request?
This PR fixes two robustness issues in `FileSystemPersistenceEngine`, which
backs the `FILESYSTEM` recovery mode of the standalone Master.
1. `serializeIntoFile` keeps the raw `FileOutputStream` as `rawOut` and
closes the serialization stream, the codec stream and `rawOut`, in this order,
each guaranteed by a nested `Utils.tryWithSafeFinally`. The close order from
SPARK-46205, where the serializer and the codec flush before the file closes,
is unchanged. The codec stream is now created inside the `tryWithSafeFinally`
body (as SPARK-60419 does on the read side), so the raw stream is closed even
when the codec stream const [...]
2. `serializeIntoFile` deletes the file and rethrows when writing fails
after it created the file. It never deletes an existing file: `File already
exists` and `Could not create file` are thrown before the file is created.
3. `read` skips a file that cannot be deserialized instead of failing as a
whole. It logs a warning with the path and the exception, and renames the file
to `corrupted_<name>`, which no longer matches the `app_`, `driver_` or
`worker_` prefix and is kept for inspection.
Items 2 and 3 together are the smallest design that unblocks recovery in
all cases:
- Deleting on write failures alone does not cover a crash or a power loss
in the middle of a write, or files that released versions already left behind.
- Skipping on read alone keeps a failed write's file under its real name
until the next start, which then warns about it and renames it.
- Writing to a temporary file, then `fsync` and `ATOMIC_MOVE`, would add an
`fsync` to every app, driver and worker registration on the Master's RPC
thread, which is slow on an NFS-mounted recovery directory. It also needs
cleanup of leftover temporary files, `ATOMIC_MOVE` support from the file
system, and a directory `fsync` that Java cannot do portably, and it still does
not cover files that are already broken. Durability is out of the scope of this
PR.
Unlike `ZooKeeperPersistenceEngine`, which deletes unreadable entries,
`read` renames them. This keeps the data when the cause is not a broken file,
e.g. an incompatible class, a changed `spark.deploy.recoveryCompressionCodec`,
or a transient I/O error.
SPARK-60419 (https://github.com/apache/spark/pull/59670) fixed the double
open in `deserializeFromFile`. This PR keeps that method as is, and fixes the
write side and the per-file handling in `read`.
### Why are the changes needed?
**Stream leaks on write failures.** When the last write fails during close,
e.g. with `ENOSPC` or `EDQUOT`, `out.close()` throws.
`ObjectOutputStream.close()` calls `flush()` first, which pushes the buffered
data through the codec, so it never reaches the codec stream's `close()`. The
`finally` block then skips `fileOut.close()`:
- The raw `FileOutputStream` stays open until GC, with or without a codec.
- With zstd, `ZstdOutputStreamNoFinalizer` is never closed, so its native
`CStream` is never freed. The class has no finalizer as a fallback.
Reaching `fileOut.close()` is not enough. When their final write fails, the
`close()` methods of `LZ4BlockOutputStream`, `ZstdOutputStreamNoFinalizer` and
the non-parallel `LZFOutputStream` throw before closing the stream they wrap.
So does `SnappyOutputStream` in snappy-java 1.1.10.8 and older, which
`branch-4.x`, `branch-4.1` and `branch-4.0` use; snappy-java 1.1.10.9 fixed it
in https://github.com/xerial/snappy-java/pull/744. Nothing else referenced the
raw `FileOutputStream`. The [...]
**Partially written files block recovery.** `serializeIntoFile` creates the
file before writing it, never deletes it on failure, and does not `fsync`.
`read` fails as a whole when any file cannot be deserialized. So an empty or
truncated `app_`, `driver_` or `worker_` file, left by a failed write, a crash
in the middle of a write, or a power loss, makes `readPersistedData` throw in
the `ElectedLeader` handler on the next start. The RPC inbox only logs
`Ignoring error`, the Master stay [...]
### Does this PR introduce _any_ user-facing change?
Yes, in the `FILESYSTEM` recovery mode, compared with the released versions:
- A failed persist no longer leaves an empty or truncated file in the
recovery directory, and no longer leaks its file descriptor.
- An unreadable file in the recovery directory no longer keeps the Master
in `STANDBY`. The Master logs a warning, renames the file to
`corrupted_<name>`, and recovers the other apps, drivers and workers. This also
applies when the recovery directory was not cleared after changing
`spark.deploy.recoveryMode` or `spark.deploy.recoveryCompressionCodec`, which
the documentation asks users to do.
### How was this patch tested?
Pass the CIs with the newly added test cases, which fail without the main
code change:
- `FileSystemPersistenceEngine closes and deletes the file when persist
fails` passes a test-only `CompressionCodec` through the engine's `codec`
constructor parameter. Its output stream fails on `flush()` and `close()`
without closing the stream it wraps, as the lz4, zstd and lzf streams do on a
full disk, or it fails to be created. The test checks that the codec stream's
`close()` is called, that the raw `FileOutputStream` is closed (`getFD.valid()`
is `false`), and that no file is [...]
- `FileSystemPersistenceEngine skips and renames unreadable files` writes
an empty file and a truncated file next to a valid one. `read` returns only the
valid object, the two files are renamed to `corrupted_<name>`, and their names
can be persisted again. Without the fix, `read` throws `java.io.EOFException`.
- The existing `SPARK-46191` test case also checks that a persist failing
with `File already exists` keeps the existing file.
```
$ build/sbt 'core/testOnly *PersistenceEngineSuite *RecoverySuite'
...
[info] Tests: succeeded 28, failed 0, canceled 0, ignored 0, pending 0
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59709 from dongjoon-hyun/SPARK-60467.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit eacf31bf9a429d28e34ebe20ad7c7444c3d026bd)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../master/FileSystemPersistenceEngine.scala | 59 ++++++++++++++++----
.../deploy/master/PersistenceEngineSuite.scala | 64 ++++++++++++++++++++++
2 files changed, 112 insertions(+), 11 deletions(-)
diff --git
a/core/src/main/scala/org/apache/spark/deploy/master/FileSystemPersistenceEngine.scala
b/core/src/main/scala/org/apache/spark/deploy/master/FileSystemPersistenceEngine.scala
index f2323048618b..ddca2cc10cd0 100644
---
a/core/src/main/scala/org/apache/spark/deploy/master/FileSystemPersistenceEngine.scala
+++
b/core/src/main/scala/org/apache/spark/deploy/master/FileSystemPersistenceEngine.scala
@@ -21,6 +21,7 @@ import java.io._
import java.nio.file.{FileAlreadyExistsException, Files, Paths}
import scala.reflect.ClassTag
+import scala.util.control.NonFatal
import org.apache.spark.internal.{Logging, MDC}
import org.apache.spark.internal.LogKeys._
@@ -63,24 +64,60 @@ private[master] class FileSystemPersistenceEngine(
override def read[T: ClassTag](prefix: String): Seq[T] = {
val files = new File(dir).listFiles().filter(_.getName.startsWith(prefix))
- files.map(deserializeFromFile[T]).toImmutableArraySeq
+ files.flatMap { file =>
+ try {
+ Some(deserializeFromFile[T](file))
+ } catch {
+ case NonFatal(e) =>
+ // Skip an unreadable file, e.g. one left truncated by a crash, so
that it cannot block
+ // recovery. Rename it, instead of deleting it, so that it is kept
for inspection but no
+ // longer matches the app_, driver_ or worker_ prefix.
+ val corrupted = new File(dir, "corrupted_" + file.getName)
+ logWarning(log"Exception while reading persisted file ${MDC(PATH,
file.getPath())}, " +
+ log"renaming it to ${MDC(NEW_PATH, corrupted.getPath())}", e)
+ if (!file.renameTo(corrupted)) {
+ logWarning(log"Error renaming ${MDC(PATH, file.getPath())}")
+ }
+ None
+ }
+ }.toImmutableArraySeq
}
private def serializeIntoFile(file: File, value: AnyRef): Unit = {
if (file.exists()) { throw new IllegalStateException("File already exists:
" + file) }
val created = file.createNewFile()
if (!created) { throw new IllegalStateException("Could not create file: "
+ file) }
- var fileOut: OutputStream = new FileOutputStream(file)
- codec.foreach { c => fileOut = c.compressedOutputStream(fileOut) }
- var out: SerializationStream = null
- Utils.tryWithSafeFinally {
- out = serializer.newInstance().serializeStream(fileOut)
- out.writeObject(value)
- } {
- if (out != null) {
- out.close()
+ try {
+ val rawOut = new FileOutputStream(file)
+ var fileOut: OutputStream = rawOut
+ var out: SerializationStream = null
+ Utils.tryWithSafeFinally {
+ codec.foreach { c => fileOut = c.compressedOutputStream(fileOut) }
+ out = serializer.newInstance().serializeStream(fileOut)
+ out.writeObject(value)
+ } {
+ // Close the serializer and the codec before the file so that they
flush first. Close
+ // each one even if the previous close fails, e.g. on a full disk,
because some codec
+ // streams do not close the stream they wrap when their final write
fails.
+ Utils.tryWithSafeFinally {
+ Utils.tryWithSafeFinally {
+ if (out != null) {
+ out.close()
+ }
+ } {
+ fileOut.close()
+ }
+ } {
+ rawOut.close()
+ }
}
- fileOut.close()
+ } catch {
+ case e: Throwable =>
+ // Remove the file created above so that a partially written file
cannot block recovery.
+ if (!file.delete()) {
+ logWarning(log"Error deleting ${MDC(PATH, file.getPath())}")
+ }
+ throw e
}
}
diff --git
a/core/src/test/scala/org/apache/spark/deploy/master/PersistenceEngineSuite.scala
b/core/src/test/scala/org/apache/spark/deploy/master/PersistenceEngineSuite.scala
index 6839afdeeff8..f594040d2e57 100644
---
a/core/src/test/scala/org/apache/spark/deploy/master/PersistenceEngineSuite.scala
+++
b/core/src/test/scala/org/apache/spark/deploy/master/PersistenceEngineSuite.scala
@@ -18,6 +18,7 @@
package org.apache.spark.deploy.master
+import java.io.{File, FileOutputStream, FilterOutputStream, InputStream,
IOException, OutputStream}
import java.net.ServerSocket
import java.nio.file.{Files, Paths}
import java.util.concurrent.ThreadLocalRandom
@@ -61,6 +62,8 @@ class PersistenceEngineSuite extends SparkFunSuite {
engine.persist("test_1", "test_1_value")
}.getMessage
assert(m.contains("File already exists"))
+ // The failed persist does not delete the existing file.
+ assert(engine.read[String]("test_1") === Seq("test_1_value"))
}
}
@@ -115,6 +118,42 @@ class PersistenceEngineSuite extends SparkFunSuite {
}
}
+ test("SPARK-60467: FileSystemPersistenceEngine closes and deletes the file
when persist fails") {
+ Seq(false, true).foreach { failOnCreate =>
+ withTempDir { dir =>
+ val codec = new FailingCompressionCodec(failOnCreate)
+ val engine = new FileSystemPersistenceEngine(
+ dir.getAbsolutePath, new JavaSerializer(new SparkConf()),
Some(codec))
+ intercept[IOException] {
+ engine.persist("test_1", "test_1_value")
+ }
+ // The codec stream is closed if it was created, and the file stream
is closed although
+ // the codec stream does not close it.
+ assert(codec.closeCalled === !failOnCreate)
+ assert(!codec.fileOut.getFD.valid())
+ // No partially written file is left behind.
+ assert(dir.list().isEmpty)
+ }
+ }
+ }
+
+ test("SPARK-60467: FileSystemPersistenceEngine skips and renames unreadable
files") {
+ withTempDir { dir =>
+ val engine = new FileSystemPersistenceEngine(
+ dir.getAbsolutePath, new JavaSerializer(new SparkConf()))
+ engine.persist("test_1", "test_1_value")
+ // An empty file and a truncated file, as a crash in the middle of a
persist can leave.
+ val bytes = Files.readAllBytes(new File(dir, "test_1").toPath)
+ Files.write(new File(dir, "test_2").toPath, Array.emptyByteArray)
+ Files.write(new File(dir, "test_3").toPath, bytes.take(bytes.length / 2))
+ assert(engine.read[String]("test_") === Seq("test_1_value"))
+ assert(dir.list().toSet === Set("test_1", "corrupted_test_2",
"corrupted_test_3"))
+ // The renamed files neither block reads nor persisting the same names
again.
+ engine.persist("test_2", "test_2_value")
+ assert(engine.read[String]("test_").toSet === Set("test_1_value",
"test_2_value"))
+ }
+ }
+
test("ZooKeeperPersistenceEngine") {
val conf = new SparkConf()
// TestingServer logs the port conflict exception rather than throwing an
exception.
@@ -199,4 +238,29 @@ class PersistenceEngineSuite extends SparkFunSuite {
(null, trialPort)
}, conf)._2
}
+
+ /**
+ * A codec that fails to create its output stream, or whose output stream
fails to flush and
+ * close, as on a full disk, without closing the file stream it wraps.
+ */
+ private class FailingCompressionCodec(failOnCreate: Boolean) extends
CompressionCodec {
+ var fileOut: FileOutputStream = _
+ var closeCalled = false
+
+ override def compressedOutputStream(s: OutputStream): OutputStream = {
+ fileOut = s.asInstanceOf[FileOutputStream]
+ if (failOnCreate) {
+ throw new IOException("Cannot create the compressed stream")
+ }
+ new FilterOutputStream(s) {
+ override def flush(): Unit = throw new IOException("No space left on
device")
+ override def close(): Unit = {
+ closeCalled = true
+ throw new IOException("No space left on device")
+ }
+ }
+ }
+
+ override def compressedInputStream(s: InputStream): InputStream = s
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]