This is an automated email from the ASF dual-hosted git repository.

LuciferYang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new a519c97a2c02 [SPARK-58461][CORE] Convert unreachable 
`_LEGACY_ERROR_TEMP_3021-3042` invariants in `SparkCoreErrors` to internal 
errors
a519c97a2c02 is described below

commit a519c97a2c02f8760f51526fd31d86411e23183f
Author: YangJie <[email protected]>
AuthorDate: Tue Aug 4 13:10:37 2026 +0800

    [SPARK-58461][CORE] Convert unreachable `_LEGACY_ERROR_TEMP_3021-3042` 
invariants in `SparkCoreErrors` to internal errors
    
    ### What changes were proposed in this pull request?
    
    This PR converts 11 legacy error conditions in `SparkCoreErrors` that guard 
**engine-internal invariants** into `SparkException.internalError(...)`, and 
deletes their entries from `error-conditions.json`. Per the error-conditions 
[README](https://github.com/apache/spark/blob/master/common/utils/src/main/resources/error/README.md),
 a throw site that a correct Spark cannot reach from user code should become an 
internal error rather than being given a user-facing name.
    
    | Legacy | Builder | Now |
    |---|---|---|
    | `_LEGACY_ERROR_TEMP_3024` | `accessNonExistentAccumulatorError` | 
`INTERNAL_ERROR` |
    | `_3025` | `sendResubmittedTaskStatusForShuffleMapStagesOnlyError` | 
`INTERNAL_ERROR` |
    | `_3030` | `taskHasNotLockedBlockError` | `INTERNAL_ERROR_STORAGE` |
    | `_3031` | `blockDoesNotExistError` | `INTERNAL_ERROR_STORAGE` |
    | `_3032` | `waitingForReplicationToFinishError` | `INTERNAL_ERROR` (keeps 
`cause`) |
    | `_3034` | `waitingForAsyncReregistrationError` | `INTERNAL_ERROR` (keeps 
`cause`) |
    | `_3038` | `failToGetBlockWithLockError` | `INTERNAL_ERROR_STORAGE` |
    | `_3039` | `blockStatusQueryReturnedNullError` | `INTERNAL_ERROR_STORAGE` |
    | `_3040` | `unexpectedBlockManagerMasterEndpointResultError` | 
`INTERNAL_ERROR_STORAGE` |
    | `_3042` | `failToGetNonShuffleBlockError` | `INTERNAL_ERROR_STORAGE` 
(keeps `cause`) |
    | `_3041` | `unsupportedOperationError` (deleted) | 
`UNSUPPORTED_CALL.WITHOUT_SUGGESTION` |
    
    Message text is preserved for all of them (a trailing period was added 
where missing). Two entries get more than a mechanical rewrite:
    
    - **`_3041` had an empty message** (`"message": [""]`), so it rendered as 
the empty string. Its builder is deleted and `DiskBlockObjectWriter.write(b: 
Int)` — an `OutputStream` method the class does not implement — now throws the 
existing `UNSUPPORTED_CALL.WITHOUT_SUGGESTION` with explicit 
`className`/`methodName`. The no-arg `SparkUnsupportedOperationException()` 
helper was tried first and rejected: it derives those from `stackTrace(3)`, 
which for a plain Scala class method resolves  [...]
    - **`_3040` had no message parameters at all** ("BlockManagerMasterEndpoint 
returned false, expected true."), so the log never said which RPC failed. Its 
builder gains a `message: Any` parameter. `BlockManagerMaster.tell` is 
`private` with two call sites, passing `RemoveExecutor(execId)` and 
`StopBlockManagerMaster`; both are a case class/object whose `toString` carries 
no host, credential, or block content (`BlockManagerId` never reaches `tell`).
    
    `_3042` uses `new SparkException("INTERNAL_ERROR_STORAGE", ..., cause = e)` 
directly because `SparkException.internalError` has no overload accepting both 
a `cause` and a `category` (precedent: `ExecutorDeadException`). 
`_3032`/`_3034` keep bare `INTERNAL_ERROR` — they are dead branches (see 
below), so the category buys nothing, and adding the missing overload to 
`common/utils` would be unrelated churn here.
    
    #### Reachability, per condition
    
    The classification is the substance of this PR, so it is spelled out rather 
than asserted:
    
    - **Unreachable by construction.** `_3025` — the only producer of 
`Resubmitted` is `TaskSetManager.executorLost`, gated on `isShuffleMapTasks`, 
so the non-`ShuffleMapStage` branch cannot be taken. `_3039` — 
`Future.sequence` cannot yield `null`; the check is a vestige of a 
pre-`awaitResult` rewrite. `_3040` — both `tell` handlers reply `true` 
unconditionally; an endpoint failure throws out of `askSync` instead. `_3042` — 
every `blockId` reaching `throwFetchFailedException` comes from  [...]
    - **Dead catch blocks.** `_3032` and `_3034` wrap only a 
`ThreadUtils.awaitReady(future, Duration.Inf)` call in `catch NonFatal`. 
`Awaitable.ready` discards the `Try` and never surfaces the future's failure, 
and `Duration.Inf` takes the `acquireSharedInterruptibly` path so no timeout is 
possible — nothing can reach either catch. (`InterruptedException` can escape 
`awaitReady`, but it is not `NonFatal`, so it passes through untouched both 
before and after this change.) `_3034`'s enclos [...]
    - **Reachable only by a benign race, and swallowed.** `_3024` is *not* 
strictly unreachable: `AccumulatorContext.get` holds `WeakReference`s and 
`ContextCleaner.doCleanupAccum` removes entries, so an in-flight 
`CompletionEvent` can carry an id that has already been collected or cleaned. 
The throw is caught by a `catch NonFatal` in the *same* method 
(`DAGScheduler.updateAccumulators`), which only logs — it never escapes to a 
job or a user. There is no user-visible exception to name, so [...]
    - **Reachable only during shutdown.** `_3031` and `_3038` are guarded by 
lock ownership at every caller, with one residual path: 
`BlockInfoManager.clear()` uses `tryLock` and drops entries without honoring 
existing holders, so an executor stopping while tasks are still finishing can 
reach them. That exception is itself suppressed by `Executor`'s `case t: 
Throwable if env.isStopped` and never reaches the driver.
    
    The remaining 10 conditions in the `_3021-3042` range are **intentionally 
left alone** — they are operational or user-reachable failures that deserve 
proper names in a follow-up, and labeling them `INTERNAL_ERROR` would report 
real failures as Spark bugs: `_3021`/`_3022` (RPC races during 
executor/endpoint shutdown), `_3023` (`submitMapStage` on a 0-partition RDD — 
user API misuse), `_3026` (`TaskInfo.duration` on an unfinished task — 
`TaskInfo` is `DeveloperApi`), `_3028` (barrier st [...]
    
    The ten converted builders remain thin wrappers in `SparkCoreErrors` rather 
than being inlined at their throw sites. Inlining would match what most of 
`core` does for internal errors, but it would enlarge the diff across five more 
files for no behavior change; `_3041` is inlined only because its builder 
became empty.
    
    ### Why are the changes needed?
    
    The error-conditions README disallows new `_LEGACY_ERROR_TEMP_*` entries 
and asks existing ones to be resolved. This clears 11 of them, under the 
umbrella [SPARK-37935](https://issues.apache.org/jira/browse/SPARK-37935). Two 
of the 11 were also defective in their own right: `_3041`'s message was empty 
and `_3040`'s carried no parameters.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No new user-facing API, but one behavior change worth a reviewer's explicit 
sign-off.
    
    `DAGScheduler.abortStage` passes a task exception straight to the user only 
when `getCondition != null && !isInternalError(condition)`. The old 
`_LEGACY_ERROR_TEMP_*` names passed that filter; `INTERNAL_ERROR*` does not. So 
for the four conditions thrown inside tasks on executors — `_3030`, `_3031`, 
`_3038`, `_3042` — a job that fails through them (after the task retries to 
`maxTaskFailures`) now reports `SparkException: Job aborted due to stage 
failure: ...` wrapping the original, in [...]
    
    `_3041`'s rendered message changes from the empty string to `Cannot call 
the method "write" of the class 
"org.apache.spark.storage.DiskBlockObjectWriter".`, and its exception type 
stays a subclass of `UnsupportedOperationException`.
    
    ### How was this patch tested?
    
    - `SparkThrowableSuite` — validates that the JSON is well-formed, 
key-sorted, and round-trips after the 11 deletions.
    - `BlockInfoManagerSuite` — its "removing a non-existent block" test was 
tightened from a bare `intercept[SparkException]` to `checkError` on 
`INTERNAL_ERROR_STORAGE`, so the conversion is now pinned by an assertion 
rather than passing vacuously.
    - `DiskBlockObjectWriterSuite` — new test asserting 
`UNSUPPORTED_CALL.WITHOUT_SUGGESTION` with the expected 
`className`/`methodName`.
    - `BlockManagerMasterSuite` — covers the `_3040` signature change.
    
    All four suites pass (70 tests) on top of the current master, and 
`core/compile` is clean. No new tests were added for the unreachable 
conversions, since there is no way to trigger them.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Opus 4.8)
    
    Closes #57663 from LuciferYang/assign-name-legacy-3021-3042.
    
    Authored-by: YangJie <[email protected]>
    Signed-off-by: yangjie01 <[email protected]>
---
 .../src/main/resources/error/error-conditions.json | 55 -----------------
 .../org/apache/spark/errors/SparkCoreErrors.scala  | 70 +++++++---------------
 .../apache/spark/storage/BlockManagerMaster.scala  |  2 +-
 .../spark/storage/DiskBlockObjectWriter.scala      |  9 ++-
 .../spark/storage/BlockInfoManagerSuite.scala      |  9 ++-
 .../spark/storage/DiskBlockObjectWriterSuite.scala | 14 ++++-
 6 files changed, 47 insertions(+), 112 deletions(-)

diff --git a/common/utils/src/main/resources/error/error-conditions.json 
b/common/utils/src/main/resources/error/error-conditions.json
index e7fc8c95a49b..a0dd28963bfb 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -11433,16 +11433,6 @@
       "Can't run submitMapStage on RDD with 0 partitions"
     ]
   },
-  "_LEGACY_ERROR_TEMP_3024" : {
-    "message" : [
-      "attempted to access non-existent accumulator <id>"
-    ]
-  },
-  "_LEGACY_ERROR_TEMP_3025" : {
-    "message" : [
-      "TaskSetManagers should only send Resubmitted task statuses for tasks in 
ShuffleMapStages."
-    ]
-  },
   "_LEGACY_ERROR_TEMP_3026" : {
     "message" : [
       "duration() called on unfinished task"
@@ -11458,31 +11448,11 @@
       "Exiting due to error from cluster scheduler: <message>"
     ]
   },
-  "_LEGACY_ERROR_TEMP_3030" : {
-    "message" : [
-      "Task <currentTaskAttemptId> has not locked block <blockId> for writing"
-    ]
-  },
-  "_LEGACY_ERROR_TEMP_3031" : {
-    "message" : [
-      "Block <blockId> does not exist"
-    ]
-  },
-  "_LEGACY_ERROR_TEMP_3032" : {
-    "message" : [
-      "Error occurred while waiting for replication to finish"
-    ]
-  },
   "_LEGACY_ERROR_TEMP_3033" : {
     "message" : [
       "Unable to register with external shuffle server due to : <message>"
     ]
   },
-  "_LEGACY_ERROR_TEMP_3034" : {
-    "message" : [
-      "Error occurred while waiting for async. reregistration"
-    ]
-  },
   "_LEGACY_ERROR_TEMP_3035" : {
     "message" : [
       "Unexpected shuffle block <blockId> with unsupported shuffle resolver 
<shuffleBlockResolver>"
@@ -11498,31 +11468,6 @@
       "Block <blockId> was not found even though it's read-locked"
     ]
   },
-  "_LEGACY_ERROR_TEMP_3038" : {
-    "message" : [
-      "get() failed for block <blockId> even though we held a lock"
-    ]
-  },
-  "_LEGACY_ERROR_TEMP_3039" : {
-    "message" : [
-      "BlockManager returned null for BlockStatus query: <blockId>"
-    ]
-  },
-  "_LEGACY_ERROR_TEMP_3040" : {
-    "message" : [
-      "BlockManagerMasterEndpoint returned false, expected true."
-    ]
-  },
-  "_LEGACY_ERROR_TEMP_3041" : {
-    "message" : [
-      ""
-    ]
-  },
-  "_LEGACY_ERROR_TEMP_3042" : {
-    "message" : [
-      "Failed to get block <blockId>, which is not a shuffle block"
-    ]
-  },
   "_LEGACY_ERROR_TEMP_3052" : {
     "message" : [
       "Unexpected resolved action: <other>"
diff --git a/core/src/main/scala/org/apache/spark/errors/SparkCoreErrors.scala 
b/core/src/main/scala/org/apache/spark/errors/SparkCoreErrors.scala
index 4813f3f94107..873254e491f2 100644
--- a/core/src/main/scala/org/apache/spark/errors/SparkCoreErrors.scala
+++ b/core/src/main/scala/org/apache/spark/errors/SparkCoreErrors.scala
@@ -248,15 +248,12 @@ private[spark] object SparkCoreErrors {
   }
 
   def accessNonExistentAccumulatorError(id: Long): Throwable = {
-    new SparkException(
-      errorClass = "_LEGACY_ERROR_TEMP_3024", messageParameters = Map("id" -> 
s"$id"), cause = null
-    )
+    SparkException.internalError(s"Attempted to access non-existent 
accumulator $id.")
   }
 
   def sendResubmittedTaskStatusForShuffleMapStagesOnlyError(): Throwable = {
-    new SparkException(
-      errorClass = "_LEGACY_ERROR_TEMP_3025", messageParameters = Map.empty, 
cause = null
-    )
+    SparkException.internalError(
+      "TaskSetManagers should only send Resubmitted task statuses for tasks in 
ShuffleMapStages.")
   }
 
   def nonEmptyEventQueueAfterTimeoutError(timeoutMillis: Long): Throwable = {
@@ -292,22 +289,13 @@ private[spark] object SparkCoreErrors {
   }
 
   def taskHasNotLockedBlockError(currentTaskAttemptId: Long, blockId: 
BlockId): Throwable = {
-    new SparkException(
-      errorClass = "_LEGACY_ERROR_TEMP_3030",
-      messageParameters = Map(
-        "currentTaskAttemptId" -> s"$currentTaskAttemptId",
-        "blockId" -> s"$blockId"
-      ),
-      cause = null
-    )
+    SparkException.internalError(
+      s"Task $currentTaskAttemptId has not locked block $blockId for writing.",
+      category = "STORAGE")
   }
 
   def blockDoesNotExistError(blockId: BlockId): Throwable = {
-    new SparkException(
-      errorClass = "_LEGACY_ERROR_TEMP_3031",
-      messageParameters = Map("blockId" -> s"$blockId"),
-      cause = null
-    )
+    SparkException.internalError(s"Block $blockId does not exist.", category = 
"STORAGE")
   }
 
   def cannotSaveBlockOnDecommissionedExecutorError(blockId: BlockId): 
Throwable = {
@@ -315,9 +303,7 @@ private[spark] object SparkCoreErrors {
   }
 
   def waitingForReplicationToFinishError(e: Throwable): Throwable = {
-    new SparkException(
-      errorClass = "_LEGACY_ERROR_TEMP_3032", messageParameters = Map.empty, 
cause = e
-    )
+    SparkException.internalError("Error occurred while waiting for replication 
to finish.", e)
   }
 
   def unableToRegisterWithExternalShuffleServerError(e: Throwable): Throwable 
= {
@@ -329,9 +315,7 @@ private[spark] object SparkCoreErrors {
   }
 
   def waitingForAsyncReregistrationError(e: Throwable): Throwable = {
-    new SparkException(
-      errorClass = "_LEGACY_ERROR_TEMP_3034", messageParameters = Map.empty, 
cause = e
-    )
+    SparkException.internalError("Error occurred while waiting for async. 
reregistration.", e)
   }
 
   def unexpectedShuffleBlockWithUnsupportedResolverError(
@@ -371,13 +355,8 @@ private[spark] object SparkCoreErrors {
   }
 
   def failToGetBlockWithLockError(blockId: BlockId): Throwable = {
-    new SparkException(
-      errorClass = "_LEGACY_ERROR_TEMP_3038",
-      messageParameters = Map(
-        "blockId" -> s"$blockId"
-      ),
-      cause = null
-    )
+    SparkException.internalError(
+      s"get() failed for block $blockId even though we held a lock.", category 
= "STORAGE")
   }
 
   def blockNotFoundError(blockId: BlockId): Throwable = {
@@ -389,17 +368,14 @@ private[spark] object SparkCoreErrors {
   }
 
   def blockStatusQueryReturnedNullError(blockId: BlockId): Throwable = {
-    new SparkException(
-      errorClass = "_LEGACY_ERROR_TEMP_3039",
-      messageParameters = Map("blockId" -> s"$blockId"),
-      cause = null
-    )
+    SparkException.internalError(
+      s"BlockManager returned null for BlockStatus query: $blockId.", category 
= "STORAGE")
   }
 
-  def unexpectedBlockManagerMasterEndpointResultError(): Throwable = {
-    new SparkException(
-      errorClass = "_LEGACY_ERROR_TEMP_3040", messageParameters = Map.empty, 
cause = null
-    )
+  def unexpectedBlockManagerMasterEndpointResultError(message: Any): Throwable 
= {
+    SparkException.internalError(
+      s"BlockManagerMasterEndpoint returned false for message $message, 
expected true.",
+      category = "STORAGE")
   }
 
   def failToCreateDirectoryError(path: String, maxAttempts: Int): Throwable = {
@@ -407,10 +383,6 @@ private[spark] object SparkCoreErrors {
       s"Failed to create directory ${path} with permission 770 after 
$maxAttempts attempts!")
   }
 
-  def unsupportedOperationError(): Throwable = {
-    new SparkUnsupportedOperationException("_LEGACY_ERROR_TEMP_3041")
-  }
-
   def noSuchElementError(): Throwable = {
     new NoSuchElementException()
   }
@@ -428,10 +400,10 @@ private[spark] object SparkCoreErrors {
 
   def failToGetNonShuffleBlockError(blockId: BlockId, e: Throwable): Throwable 
= {
     new SparkException(
-      errorClass = "_LEGACY_ERROR_TEMP_3042",
-      messageParameters = Map("blockId" -> s"$blockId"),
-      cause = e
-    )
+      errorClass = "INTERNAL_ERROR_STORAGE",
+      messageParameters = Map(
+        "message" -> s"Failed to get block $blockId, which is not a shuffle 
block."),
+      cause = e)
   }
 
   def graphiteSinkInvalidProtocolError(invalidProtocol: String): Throwable = {
diff --git 
a/core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala 
b/core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala
index 84a70d27047c..47b6e2a50525 100644
--- a/core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala
+++ b/core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala
@@ -342,7 +342,7 @@ class BlockManagerMaster(
   /** Send a one-way message to the master endpoint, to which we expect it to 
reply with true. */
   private def tell(message: Any): Unit = {
     if (!driverEndpoint.askSync[Boolean](message)) {
-      throw SparkCoreErrors.unexpectedBlockManagerMasterEndpointResultError()
+      throw 
SparkCoreErrors.unexpectedBlockManagerMasterEndpointResultError(message)
     }
   }
 
diff --git 
a/core/src/main/scala/org/apache/spark/storage/DiskBlockObjectWriter.scala 
b/core/src/main/scala/org/apache/spark/storage/DiskBlockObjectWriter.scala
index 9964e64724f6..3ede60974387 100644
--- a/core/src/main/scala/org/apache/spark/storage/DiskBlockObjectWriter.scala
+++ b/core/src/main/scala/org/apache/spark/storage/DiskBlockObjectWriter.scala
@@ -22,8 +22,7 @@ import java.nio.channels.{ClosedByInterruptException, 
FileChannel}
 import java.nio.file.Files
 import java.util.zip.Checksum
 
-import org.apache.spark.SparkException
-import org.apache.spark.errors.SparkCoreErrors
+import org.apache.spark.{SparkException, SparkUnsupportedOperationException}
 import org.apache.spark.internal.Logging
 import org.apache.spark.internal.LogKeys._
 import org.apache.spark.io.MutableCheckedOutputStream
@@ -338,7 +337,11 @@ private[spark] class DiskBlockObjectWriter(
     recordWritten()
   }
 
-  override def write(b: Int): Unit = throw 
SparkCoreErrors.unsupportedOperationError()
+  override def write(b: Int): Unit = throw new 
SparkUnsupportedOperationException(
+    errorClass = "UNSUPPORTED_CALL.WITHOUT_SUGGESTION",
+    messageParameters = Map(
+      "className" -> classOf[DiskBlockObjectWriter].getName,
+      "methodName" -> "write"))
 
   override def write(kvBytes: Array[Byte], offs: Int, len: Int): Unit = {
     if (!streamOpen) {
diff --git 
a/core/src/test/scala/org/apache/spark/storage/BlockInfoManagerSuite.scala 
b/core/src/test/scala/org/apache/spark/storage/BlockInfoManagerSuite.scala
index 4b34c13706ad..016c49c2d1a4 100644
--- a/core/src/test/scala/org/apache/spark/storage/BlockInfoManagerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/storage/BlockInfoManagerSuite.scala
@@ -313,9 +313,12 @@ class BlockInfoManagerSuite extends SparkFunSuite {
 
   test("removing a non-existent block throws SparkException") {
     withTaskId(0) {
-      intercept[SparkException] {
-        blockInfoManager.removeBlock("non-existent-block")
-      }
+      checkError(
+        exception = intercept[SparkException] {
+          blockInfoManager.removeBlock("non-existent-block")
+        },
+        condition = "INTERNAL_ERROR_STORAGE",
+        parameters = Map("message" -> "Block test_non-existent-block does not 
exist."))
     }
   }
 
diff --git 
a/core/src/test/scala/org/apache/spark/storage/DiskBlockObjectWriterSuite.scala 
b/core/src/test/scala/org/apache/spark/storage/DiskBlockObjectWriterSuite.scala
index 4352436c872f..f103f3c305cf 100644
--- 
a/core/src/test/scala/org/apache/spark/storage/DiskBlockObjectWriterSuite.scala
+++ 
b/core/src/test/scala/org/apache/spark/storage/DiskBlockObjectWriterSuite.scala
@@ -21,7 +21,7 @@ import java.nio.ByteBuffer
 
 import scala.reflect.ClassTag
 
-import org.apache.spark.{SparkConf, SparkException, SparkFunSuite}
+import org.apache.spark.{SparkConf, SparkException, SparkFunSuite, 
SparkUnsupportedOperationException}
 import org.apache.spark.executor.ShuffleWriteMetrics
 import org.apache.spark.serializer.{DeserializationStream, JavaSerializer, 
SerializationStream, Serializer, SerializerInstance, SerializerManager}
 import org.apache.spark.util.Utils
@@ -214,6 +214,18 @@ class DiskBlockObjectWriterSuite extends SparkFunSuite {
     assert(bs.isClosed)
     assert(objOut.isClosed)
   }
+
+  test("write(b: Int) is not supported") {
+    val (writer, _, _) = createWriter()
+    checkError(
+      exception = intercept[SparkUnsupportedOperationException] {
+        writer.write(1)
+      },
+      condition = "UNSUPPORTED_CALL.WITHOUT_SUGGESTION",
+      parameters = Map(
+        "className" -> "org.apache.spark.storage.DiskBlockObjectWriter",
+        "methodName" -> "write"))
+  }
 }
 
 trait CloseDetecting {


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to