This is an automated email from the ASF dual-hosted git repository.
LuciferYang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new a519c97a2c02 [SPARK-58461][CORE] Convert unreachable
`_LEGACY_ERROR_TEMP_3021-3042` invariants in `SparkCoreErrors` to internal
errors
a519c97a2c02 is described below
commit a519c97a2c02f8760f51526fd31d86411e23183f
Author: YangJie <[email protected]>
AuthorDate: Tue Aug 4 13:10:37 2026 +0800
[SPARK-58461][CORE] Convert unreachable `_LEGACY_ERROR_TEMP_3021-3042`
invariants in `SparkCoreErrors` to internal errors
### What changes were proposed in this pull request?
This PR converts 11 legacy error conditions in `SparkCoreErrors` that guard
**engine-internal invariants** into `SparkException.internalError(...)`, and
deletes their entries from `error-conditions.json`. Per the error-conditions
[README](https://github.com/apache/spark/blob/master/common/utils/src/main/resources/error/README.md),
a throw site that a correct Spark cannot reach from user code should become an
internal error rather than being given a user-facing name.
| Legacy | Builder | Now |
|---|---|---|
| `_LEGACY_ERROR_TEMP_3024` | `accessNonExistentAccumulatorError` |
`INTERNAL_ERROR` |
| `_3025` | `sendResubmittedTaskStatusForShuffleMapStagesOnlyError` |
`INTERNAL_ERROR` |
| `_3030` | `taskHasNotLockedBlockError` | `INTERNAL_ERROR_STORAGE` |
| `_3031` | `blockDoesNotExistError` | `INTERNAL_ERROR_STORAGE` |
| `_3032` | `waitingForReplicationToFinishError` | `INTERNAL_ERROR` (keeps
`cause`) |
| `_3034` | `waitingForAsyncReregistrationError` | `INTERNAL_ERROR` (keeps
`cause`) |
| `_3038` | `failToGetBlockWithLockError` | `INTERNAL_ERROR_STORAGE` |
| `_3039` | `blockStatusQueryReturnedNullError` | `INTERNAL_ERROR_STORAGE` |
| `_3040` | `unexpectedBlockManagerMasterEndpointResultError` |
`INTERNAL_ERROR_STORAGE` |
| `_3042` | `failToGetNonShuffleBlockError` | `INTERNAL_ERROR_STORAGE`
(keeps `cause`) |
| `_3041` | `unsupportedOperationError` (deleted) |
`UNSUPPORTED_CALL.WITHOUT_SUGGESTION` |
Message text is preserved for all of them (a trailing period was added
where missing). Two entries get more than a mechanical rewrite:
- **`_3041` had an empty message** (`"message": [""]`), so it rendered as
the empty string. Its builder is deleted and `DiskBlockObjectWriter.write(b:
Int)` — an `OutputStream` method the class does not implement — now throws the
existing `UNSUPPORTED_CALL.WITHOUT_SUGGESTION` with explicit
`className`/`methodName`. The no-arg `SparkUnsupportedOperationException()`
helper was tried first and rejected: it derives those from `stackTrace(3)`,
which for a plain Scala class method resolves [...]
- **`_3040` had no message parameters at all** ("BlockManagerMasterEndpoint
returned false, expected true."), so the log never said which RPC failed. Its
builder gains a `message: Any` parameter. `BlockManagerMaster.tell` is
`private` with two call sites, passing `RemoveExecutor(execId)` and
`StopBlockManagerMaster`; both are a case class/object whose `toString` carries
no host, credential, or block content (`BlockManagerId` never reaches `tell`).
`_3042` uses `new SparkException("INTERNAL_ERROR_STORAGE", ..., cause = e)`
directly because `SparkException.internalError` has no overload accepting both
a `cause` and a `category` (precedent: `ExecutorDeadException`).
`_3032`/`_3034` keep bare `INTERNAL_ERROR` — they are dead branches (see
below), so the category buys nothing, and adding the missing overload to
`common/utils` would be unrelated churn here.
#### Reachability, per condition
The classification is the substance of this PR, so it is spelled out rather
than asserted:
- **Unreachable by construction.** `_3025` — the only producer of
`Resubmitted` is `TaskSetManager.executorLost`, gated on `isShuffleMapTasks`,
so the non-`ShuffleMapStage` branch cannot be taken. `_3039` —
`Future.sequence` cannot yield `null`; the check is a vestige of a
pre-`awaitResult` rewrite. `_3040` — both `tell` handlers reply `true`
unconditionally; an endpoint failure throws out of `askSync` instead. `_3042` —
every `blockId` reaching `throwFetchFailedException` comes from [...]
- **Dead catch blocks.** `_3032` and `_3034` wrap only a
`ThreadUtils.awaitReady(future, Duration.Inf)` call in `catch NonFatal`.
`Awaitable.ready` discards the `Try` and never surfaces the future's failure,
and `Duration.Inf` takes the `acquireSharedInterruptibly` path so no timeout is
possible — nothing can reach either catch. (`InterruptedException` can escape
`awaitReady`, but it is not `NonFatal`, so it passes through untouched both
before and after this change.) `_3034`'s enclos [...]
- **Reachable only by a benign race, and swallowed.** `_3024` is *not*
strictly unreachable: `AccumulatorContext.get` holds `WeakReference`s and
`ContextCleaner.doCleanupAccum` removes entries, so an in-flight
`CompletionEvent` can carry an id that has already been collected or cleaned.
The throw is caught by a `catch NonFatal` in the *same* method
(`DAGScheduler.updateAccumulators`), which only logs — it never escapes to a
job or a user. There is no user-visible exception to name, so [...]
- **Reachable only during shutdown.** `_3031` and `_3038` are guarded by
lock ownership at every caller, with one residual path:
`BlockInfoManager.clear()` uses `tryLock` and drops entries without honoring
existing holders, so an executor stopping while tasks are still finishing can
reach them. That exception is itself suppressed by `Executor`'s `case t:
Throwable if env.isStopped` and never reaches the driver.
The remaining 10 conditions in the `_3021-3042` range are **intentionally
left alone** — they are operational or user-reachable failures that deserve
proper names in a follow-up, and labeling them `INTERNAL_ERROR` would report
real failures as Spark bugs: `_3021`/`_3022` (RPC races during
executor/endpoint shutdown), `_3023` (`submitMapStage` on a 0-partition RDD —
user API misuse), `_3026` (`TaskInfo.duration` on an unfinished task —
`TaskInfo` is `DeveloperApi`), `_3028` (barrier st [...]
The ten converted builders remain thin wrappers in `SparkCoreErrors` rather
than being inlined at their throw sites. Inlining would match what most of
`core` does for internal errors, but it would enlarge the diff across five more
files for no behavior change; `_3041` is inlined only because its builder
became empty.
### Why are the changes needed?
The error-conditions README disallows new `_LEGACY_ERROR_TEMP_*` entries
and asks existing ones to be resolved. This clears 11 of them, under the
umbrella [SPARK-37935](https://issues.apache.org/jira/browse/SPARK-37935). Two
of the 11 were also defective in their own right: `_3041`'s message was empty
and `_3040`'s carried no parameters.
### Does this PR introduce _any_ user-facing change?
No new user-facing API, but one behavior change worth a reviewer's explicit
sign-off.
`DAGScheduler.abortStage` passes a task exception straight to the user only
when `getCondition != null && !isInternalError(condition)`. The old
`_LEGACY_ERROR_TEMP_*` names passed that filter; `INTERNAL_ERROR*` does not. So
for the four conditions thrown inside tasks on executors — `_3030`, `_3031`,
`_3038`, `_3042` — a job that fails through them (after the task retries to
`maxTaskFailures`) now reports `SparkException: Job aborted due to stage
failure: ...` wrapping the original, in [...]
`_3041`'s rendered message changes from the empty string to `Cannot call
the method "write" of the class
"org.apache.spark.storage.DiskBlockObjectWriter".`, and its exception type
stays a subclass of `UnsupportedOperationException`.
### How was this patch tested?
- `SparkThrowableSuite` — validates that the JSON is well-formed,
key-sorted, and round-trips after the 11 deletions.
- `BlockInfoManagerSuite` — its "removing a non-existent block" test was
tightened from a bare `intercept[SparkException]` to `checkError` on
`INTERNAL_ERROR_STORAGE`, so the conversion is now pinned by an assertion
rather than passing vacuously.
- `DiskBlockObjectWriterSuite` — new test asserting
`UNSUPPORTED_CALL.WITHOUT_SUGGESTION` with the expected
`className`/`methodName`.
- `BlockManagerMasterSuite` — covers the `_3040` signature change.
All four suites pass (70 tests) on top of the current master, and
`core/compile` is clean. No new tests were added for the unreachable
conversions, since there is no way to trigger them.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #57663 from LuciferYang/assign-name-legacy-3021-3042.
Authored-by: YangJie <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
---
.../src/main/resources/error/error-conditions.json | 55 -----------------
.../org/apache/spark/errors/SparkCoreErrors.scala | 70 +++++++---------------
.../apache/spark/storage/BlockManagerMaster.scala | 2 +-
.../spark/storage/DiskBlockObjectWriter.scala | 9 ++-
.../spark/storage/BlockInfoManagerSuite.scala | 9 ++-
.../spark/storage/DiskBlockObjectWriterSuite.scala | 14 ++++-
6 files changed, 47 insertions(+), 112 deletions(-)
diff --git a/common/utils/src/main/resources/error/error-conditions.json
b/common/utils/src/main/resources/error/error-conditions.json
index e7fc8c95a49b..a0dd28963bfb 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -11433,16 +11433,6 @@
"Can't run submitMapStage on RDD with 0 partitions"
]
},
- "_LEGACY_ERROR_TEMP_3024" : {
- "message" : [
- "attempted to access non-existent accumulator <id>"
- ]
- },
- "_LEGACY_ERROR_TEMP_3025" : {
- "message" : [
- "TaskSetManagers should only send Resubmitted task statuses for tasks in
ShuffleMapStages."
- ]
- },
"_LEGACY_ERROR_TEMP_3026" : {
"message" : [
"duration() called on unfinished task"
@@ -11458,31 +11448,11 @@
"Exiting due to error from cluster scheduler: <message>"
]
},
- "_LEGACY_ERROR_TEMP_3030" : {
- "message" : [
- "Task <currentTaskAttemptId> has not locked block <blockId> for writing"
- ]
- },
- "_LEGACY_ERROR_TEMP_3031" : {
- "message" : [
- "Block <blockId> does not exist"
- ]
- },
- "_LEGACY_ERROR_TEMP_3032" : {
- "message" : [
- "Error occurred while waiting for replication to finish"
- ]
- },
"_LEGACY_ERROR_TEMP_3033" : {
"message" : [
"Unable to register with external shuffle server due to : <message>"
]
},
- "_LEGACY_ERROR_TEMP_3034" : {
- "message" : [
- "Error occurred while waiting for async. reregistration"
- ]
- },
"_LEGACY_ERROR_TEMP_3035" : {
"message" : [
"Unexpected shuffle block <blockId> with unsupported shuffle resolver
<shuffleBlockResolver>"
@@ -11498,31 +11468,6 @@
"Block <blockId> was not found even though it's read-locked"
]
},
- "_LEGACY_ERROR_TEMP_3038" : {
- "message" : [
- "get() failed for block <blockId> even though we held a lock"
- ]
- },
- "_LEGACY_ERROR_TEMP_3039" : {
- "message" : [
- "BlockManager returned null for BlockStatus query: <blockId>"
- ]
- },
- "_LEGACY_ERROR_TEMP_3040" : {
- "message" : [
- "BlockManagerMasterEndpoint returned false, expected true."
- ]
- },
- "_LEGACY_ERROR_TEMP_3041" : {
- "message" : [
- ""
- ]
- },
- "_LEGACY_ERROR_TEMP_3042" : {
- "message" : [
- "Failed to get block <blockId>, which is not a shuffle block"
- ]
- },
"_LEGACY_ERROR_TEMP_3052" : {
"message" : [
"Unexpected resolved action: <other>"
diff --git a/core/src/main/scala/org/apache/spark/errors/SparkCoreErrors.scala
b/core/src/main/scala/org/apache/spark/errors/SparkCoreErrors.scala
index 4813f3f94107..873254e491f2 100644
--- a/core/src/main/scala/org/apache/spark/errors/SparkCoreErrors.scala
+++ b/core/src/main/scala/org/apache/spark/errors/SparkCoreErrors.scala
@@ -248,15 +248,12 @@ private[spark] object SparkCoreErrors {
}
def accessNonExistentAccumulatorError(id: Long): Throwable = {
- new SparkException(
- errorClass = "_LEGACY_ERROR_TEMP_3024", messageParameters = Map("id" ->
s"$id"), cause = null
- )
+ SparkException.internalError(s"Attempted to access non-existent
accumulator $id.")
}
def sendResubmittedTaskStatusForShuffleMapStagesOnlyError(): Throwable = {
- new SparkException(
- errorClass = "_LEGACY_ERROR_TEMP_3025", messageParameters = Map.empty,
cause = null
- )
+ SparkException.internalError(
+ "TaskSetManagers should only send Resubmitted task statuses for tasks in
ShuffleMapStages.")
}
def nonEmptyEventQueueAfterTimeoutError(timeoutMillis: Long): Throwable = {
@@ -292,22 +289,13 @@ private[spark] object SparkCoreErrors {
}
def taskHasNotLockedBlockError(currentTaskAttemptId: Long, blockId:
BlockId): Throwable = {
- new SparkException(
- errorClass = "_LEGACY_ERROR_TEMP_3030",
- messageParameters = Map(
- "currentTaskAttemptId" -> s"$currentTaskAttemptId",
- "blockId" -> s"$blockId"
- ),
- cause = null
- )
+ SparkException.internalError(
+ s"Task $currentTaskAttemptId has not locked block $blockId for writing.",
+ category = "STORAGE")
}
def blockDoesNotExistError(blockId: BlockId): Throwable = {
- new SparkException(
- errorClass = "_LEGACY_ERROR_TEMP_3031",
- messageParameters = Map("blockId" -> s"$blockId"),
- cause = null
- )
+ SparkException.internalError(s"Block $blockId does not exist.", category =
"STORAGE")
}
def cannotSaveBlockOnDecommissionedExecutorError(blockId: BlockId):
Throwable = {
@@ -315,9 +303,7 @@ private[spark] object SparkCoreErrors {
}
def waitingForReplicationToFinishError(e: Throwable): Throwable = {
- new SparkException(
- errorClass = "_LEGACY_ERROR_TEMP_3032", messageParameters = Map.empty,
cause = e
- )
+ SparkException.internalError("Error occurred while waiting for replication
to finish.", e)
}
def unableToRegisterWithExternalShuffleServerError(e: Throwable): Throwable
= {
@@ -329,9 +315,7 @@ private[spark] object SparkCoreErrors {
}
def waitingForAsyncReregistrationError(e: Throwable): Throwable = {
- new SparkException(
- errorClass = "_LEGACY_ERROR_TEMP_3034", messageParameters = Map.empty,
cause = e
- )
+ SparkException.internalError("Error occurred while waiting for async.
reregistration.", e)
}
def unexpectedShuffleBlockWithUnsupportedResolverError(
@@ -371,13 +355,8 @@ private[spark] object SparkCoreErrors {
}
def failToGetBlockWithLockError(blockId: BlockId): Throwable = {
- new SparkException(
- errorClass = "_LEGACY_ERROR_TEMP_3038",
- messageParameters = Map(
- "blockId" -> s"$blockId"
- ),
- cause = null
- )
+ SparkException.internalError(
+ s"get() failed for block $blockId even though we held a lock.", category
= "STORAGE")
}
def blockNotFoundError(blockId: BlockId): Throwable = {
@@ -389,17 +368,14 @@ private[spark] object SparkCoreErrors {
}
def blockStatusQueryReturnedNullError(blockId: BlockId): Throwable = {
- new SparkException(
- errorClass = "_LEGACY_ERROR_TEMP_3039",
- messageParameters = Map("blockId" -> s"$blockId"),
- cause = null
- )
+ SparkException.internalError(
+ s"BlockManager returned null for BlockStatus query: $blockId.", category
= "STORAGE")
}
- def unexpectedBlockManagerMasterEndpointResultError(): Throwable = {
- new SparkException(
- errorClass = "_LEGACY_ERROR_TEMP_3040", messageParameters = Map.empty,
cause = null
- )
+ def unexpectedBlockManagerMasterEndpointResultError(message: Any): Throwable
= {
+ SparkException.internalError(
+ s"BlockManagerMasterEndpoint returned false for message $message,
expected true.",
+ category = "STORAGE")
}
def failToCreateDirectoryError(path: String, maxAttempts: Int): Throwable = {
@@ -407,10 +383,6 @@ private[spark] object SparkCoreErrors {
s"Failed to create directory ${path} with permission 770 after
$maxAttempts attempts!")
}
- def unsupportedOperationError(): Throwable = {
- new SparkUnsupportedOperationException("_LEGACY_ERROR_TEMP_3041")
- }
-
def noSuchElementError(): Throwable = {
new NoSuchElementException()
}
@@ -428,10 +400,10 @@ private[spark] object SparkCoreErrors {
def failToGetNonShuffleBlockError(blockId: BlockId, e: Throwable): Throwable
= {
new SparkException(
- errorClass = "_LEGACY_ERROR_TEMP_3042",
- messageParameters = Map("blockId" -> s"$blockId"),
- cause = e
- )
+ errorClass = "INTERNAL_ERROR_STORAGE",
+ messageParameters = Map(
+ "message" -> s"Failed to get block $blockId, which is not a shuffle
block."),
+ cause = e)
}
def graphiteSinkInvalidProtocolError(invalidProtocol: String): Throwable = {
diff --git
a/core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala
b/core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala
index 84a70d27047c..47b6e2a50525 100644
--- a/core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala
+++ b/core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala
@@ -342,7 +342,7 @@ class BlockManagerMaster(
/** Send a one-way message to the master endpoint, to which we expect it to
reply with true. */
private def tell(message: Any): Unit = {
if (!driverEndpoint.askSync[Boolean](message)) {
- throw SparkCoreErrors.unexpectedBlockManagerMasterEndpointResultError()
+ throw
SparkCoreErrors.unexpectedBlockManagerMasterEndpointResultError(message)
}
}
diff --git
a/core/src/main/scala/org/apache/spark/storage/DiskBlockObjectWriter.scala
b/core/src/main/scala/org/apache/spark/storage/DiskBlockObjectWriter.scala
index 9964e64724f6..3ede60974387 100644
--- a/core/src/main/scala/org/apache/spark/storage/DiskBlockObjectWriter.scala
+++ b/core/src/main/scala/org/apache/spark/storage/DiskBlockObjectWriter.scala
@@ -22,8 +22,7 @@ import java.nio.channels.{ClosedByInterruptException,
FileChannel}
import java.nio.file.Files
import java.util.zip.Checksum
-import org.apache.spark.SparkException
-import org.apache.spark.errors.SparkCoreErrors
+import org.apache.spark.{SparkException, SparkUnsupportedOperationException}
import org.apache.spark.internal.Logging
import org.apache.spark.internal.LogKeys._
import org.apache.spark.io.MutableCheckedOutputStream
@@ -338,7 +337,11 @@ private[spark] class DiskBlockObjectWriter(
recordWritten()
}
- override def write(b: Int): Unit = throw
SparkCoreErrors.unsupportedOperationError()
+ override def write(b: Int): Unit = throw new
SparkUnsupportedOperationException(
+ errorClass = "UNSUPPORTED_CALL.WITHOUT_SUGGESTION",
+ messageParameters = Map(
+ "className" -> classOf[DiskBlockObjectWriter].getName,
+ "methodName" -> "write"))
override def write(kvBytes: Array[Byte], offs: Int, len: Int): Unit = {
if (!streamOpen) {
diff --git
a/core/src/test/scala/org/apache/spark/storage/BlockInfoManagerSuite.scala
b/core/src/test/scala/org/apache/spark/storage/BlockInfoManagerSuite.scala
index 4b34c13706ad..016c49c2d1a4 100644
--- a/core/src/test/scala/org/apache/spark/storage/BlockInfoManagerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/storage/BlockInfoManagerSuite.scala
@@ -313,9 +313,12 @@ class BlockInfoManagerSuite extends SparkFunSuite {
test("removing a non-existent block throws SparkException") {
withTaskId(0) {
- intercept[SparkException] {
- blockInfoManager.removeBlock("non-existent-block")
- }
+ checkError(
+ exception = intercept[SparkException] {
+ blockInfoManager.removeBlock("non-existent-block")
+ },
+ condition = "INTERNAL_ERROR_STORAGE",
+ parameters = Map("message" -> "Block test_non-existent-block does not
exist."))
}
}
diff --git
a/core/src/test/scala/org/apache/spark/storage/DiskBlockObjectWriterSuite.scala
b/core/src/test/scala/org/apache/spark/storage/DiskBlockObjectWriterSuite.scala
index 4352436c872f..f103f3c305cf 100644
---
a/core/src/test/scala/org/apache/spark/storage/DiskBlockObjectWriterSuite.scala
+++
b/core/src/test/scala/org/apache/spark/storage/DiskBlockObjectWriterSuite.scala
@@ -21,7 +21,7 @@ import java.nio.ByteBuffer
import scala.reflect.ClassTag
-import org.apache.spark.{SparkConf, SparkException, SparkFunSuite}
+import org.apache.spark.{SparkConf, SparkException, SparkFunSuite,
SparkUnsupportedOperationException}
import org.apache.spark.executor.ShuffleWriteMetrics
import org.apache.spark.serializer.{DeserializationStream, JavaSerializer,
SerializationStream, Serializer, SerializerInstance, SerializerManager}
import org.apache.spark.util.Utils
@@ -214,6 +214,18 @@ class DiskBlockObjectWriterSuite extends SparkFunSuite {
assert(bs.isClosed)
assert(objOut.isClosed)
}
+
+ test("write(b: Int) is not supported") {
+ val (writer, _, _) = createWriter()
+ checkError(
+ exception = intercept[SparkUnsupportedOperationException] {
+ writer.write(1)
+ },
+ condition = "UNSUPPORTED_CALL.WITHOUT_SUGGESTION",
+ parameters = Map(
+ "className" -> "org.apache.spark.storage.DiskBlockObjectWriter",
+ "methodName" -> "write"))
+ }
}
trait CloseDetecting {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]