This is an automated email from the ASF dual-hosted git repository.
LuciferYang pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new bda392c47889 [SPARK-58744][CORE] Assign a name to the error condition
`_LEGACY_ERROR_TEMP_3008-3010`
bda392c47889 is described below
commit bda392c47889ee9d4a66052e1956e5e896119184
Author: YangJie <[email protected]>
AuthorDate: Thu Aug 13 10:47:09 2026 +0800
[SPARK-58744][CORE] Assign a name to the error condition
`_LEGACY_ERROR_TEMP_3008-3010`
### What changes were proposed in this pull request?
This PR converts the three `_LEGACY_ERROR_TEMP_*` conditions in
`SparkCoreErrors` that reject an array-typed RDD key (`_3008`, `_3009` and
`_3010`) into a new `UNSUPPORTED_ARRAY_KEY` umbrella with three sub-conditions,
continuing the cleanup under
[SPARK-37935](https://issues.apache.org/jira/browse/SPARK-37935).
| Legacy | Builder | Now | SQLSTATE |
|---|---|---|---|
| `_LEGACY_ERROR_TEMP_3008` | `cannotUseMapSideCombiningWithArrayKeyError`
| `UNSUPPORTED_ARRAY_KEY.MAP_SIDE_COMBINE` | 0A000 |
| `_3009` | `hashPartitionerCannotPartitionArrayKeyError` |
`UNSUPPORTED_ARRAY_KEY.HASH_PARTITIONER` | 0A000 |
| `_3010` | `reduceByKeyLocallyNotSupportArrayKeysError` |
`UNSUPPORTED_ARRAY_KEY.REDUCE_BY_KEY_LOCALLY` | 0A000 |
The umbrella message is `Array keys are not supported by:` and each
sub-condition completes the sentence with the mechanism that did the rejecting
(`HashPartitioner.` / `map-side combining.` / `reduceByKeyLocally().`).
The split is by mechanism rather than by user-facing API because an
API-based split does not partition the throw sites. `HASH_PARTITIONER` covers 5
of the 7 sites, spread over `partitionBy`, `combineByKeyWithClassTag` and three
`cogroup` overloads, and those are reached from `partitionBy`, `groupByKey`,
`join`, the three outer joins, `cogroup` and `groupWith`. In the other
direction, `combineByKeyWithClassTag` alone can raise either `MAP_SIDE_COMBINE`
or `HASH_PARTITIONER` depending o [...]
The builders keep their Scala names, which keeps the diff off
`PairRDDFunctions`'s 7 throw sites.
### Reachability
All three are reachable directly from the RDD API on the driver, so they
get user-facing names rather than becoming internal errors. The guards run
eagerly in the calling method inside `withScope`, before any RDD is
constructed, so no action is needed to trigger them and no task is involved,
which keeps `DAGScheduler.abortStage`'s internal-error handling out of the
picture.
`RDD.countByValueApprox`'s own array check (`_LEGACY_ERROR_TEMP_3015`,
reached from `countByKeyApprox`) is deliberately left out even though it is the
same family. It is a different check: it tests the *element* class of the
mapped key RDD rather than a pair RDD's key class, and its message names
`countByValueApprox()` rather than the API the user called, so folding it under
`UNSUPPORTED_ARRAY_KEY` would overstate the umbrella's scope.
### Why are the changes needed?
The error-conditions
[README](https://github.com/apache/spark/blob/master/common/utils/src/main/resources/error/README.md)
disallows new `_LEGACY_ERROR_TEMP_*` entries and asks existing ones to be
resolved. This clears three of them.
The three old messages were also inconsistent with each other for one
concept: two ended with a period and one did not, and each named the mechanism
in its own phrasing (`Cannot use map-side combining ...`, `HashPartitioner
cannot partition ...`, `reduceByKeyLocally() does not support ...`). The
umbrella gives them one sentence frame.
### Does this PR introduce _any_ user-facing change?
Yes, to error messages, with no API change.
Converting any legacy condition changes the rendered string in two
mechanical ways: `SparkThrowableHelper.formatErrorMessage` suppresses the
`[CONDITION] ` prefix only for `_LEGACY_ERROR_`-prefixed names, and appends `
SQLSTATE: xxxxx` when a sqlState exists (legacy entries have none, so these
three gain both). Beyond that the message bodies change:
- `_3008`: `Cannot use map-side combining with array keys.` ->
`[UNSUPPORTED_ARRAY_KEY.MAP_SIDE_COMBINE] Array keys are not supported by:
map-side combining. SQLSTATE: 0A000`
- `_3009`: `HashPartitioner cannot partition array keys.` ->
`[UNSUPPORTED_ARRAY_KEY.HASH_PARTITIONER] Array keys are not supported by:
HashPartitioner. SQLSTATE: 0A000`
- `_3010`: `reduceByKeyLocally() does not support array keys` ->
`[UNSUPPORTED_ARRAY_KEY.REDUCE_BY_KEY_LOCALLY] Array keys are not supported by:
reduceByKeyLocally(). SQLSTATE: 0A000`
The thrown type stays `SparkException` for all three. Sibling 0A000
conditions in `SparkCoreErrors` use both `SparkException`
(`UNSUPPORTED_ADD_FILE.*`) and `SparkUnsupportedOperationException`
(`UNSUPPORTED_CALL.TASK_NOT_FINISHED`), so there is no convention to follow
here, and switching would move these out of `SparkException`'s hierarchy and
stop existing `catch` blocks from catching them.
### How was this patch tested?
`PartitioningSuite`'s `"partitioning Java arrays should fail"` already
covered all three conditions, but its helper was
`intercept[SparkException](testFun).getMessage.contains("array")`, whose
Boolean result was discarded, so all twelve assertions only checked that some
`SparkException` was thrown, not that it was about array keys. The helper now
takes the expected sub-condition and calls `checkError` with it plus SQLSTATE,
and 11 of the 12 assertions fail against the pre-change code [...]
Asserting a specific sub-condition per line pins which guard fires first,
in particular that `combineByKeyWithClassTag` checks `mapSideCombine` before it
looks at the partitioner. An inline comment records this, since reordering
those two guards would change a user-visible condition name and should be
deliberate.
The SQLSTATE assertion was verified to be live by temporarily setting the
JSON value to `42000` and watching the test fail with `sqlState: expected
'0A000' but got '42000'` before restoring it. `checkError` skips the comparison
when `sqlState` is `None`, and `SparkThrowableSuite` only checks that a state
is registered, so a wrong SQLSTATE would otherwise ship green.
Ran `core/testOnly org.apache.spark.SparkThrowableSuite
org.apache.spark.PartitioningSuite` (50 tests, all passing).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #57968 from LuciferYang/assign-name-legacy-3008-3010.
Authored-by: YangJie <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
(cherry picked from commit c528f4bbe1f323490e253c9a3d7e3f9fb559a571)
Signed-off-by: yangjie01 <[email protected]>
---
.../src/main/resources/error/error-conditions.json | 38 +++++++++++++---------
.../org/apache/spark/errors/SparkCoreErrors.scala | 12 +++++--
.../scala/org/apache/spark/PartitioningSuite.scala | 38 ++++++++++++++--------
3 files changed, 56 insertions(+), 32 deletions(-)
diff --git a/common/utils/src/main/resources/error/error-conditions.json
b/common/utils/src/main/resources/error/error-conditions.json
index db1bfd43ad23..fe811d4abe59 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -8393,6 +8393,29 @@
},
"sqlState" : "0A000"
},
+ "UNSUPPORTED_ARRAY_KEY" : {
+ "message" : [
+ "Array keys are not supported by:"
+ ],
+ "subClass" : {
+ "HASH_PARTITIONER" : {
+ "message" : [
+ "HashPartitioner."
+ ]
+ },
+ "MAP_SIDE_COMBINE" : {
+ "message" : [
+ "map-side combining."
+ ]
+ },
+ "REDUCE_BY_KEY_LOCALLY" : {
+ "message" : [
+ "reduceByKeyLocally()."
+ ]
+ }
+ },
+ "sqlState" : "0A000"
+ },
"UNSUPPORTED_ARROWTYPE" : {
"message" : [
"Unsupported arrow type <typeName>."
@@ -11474,21 +11497,6 @@
"empty RDD"
]
},
- "_LEGACY_ERROR_TEMP_3008" : {
- "message" : [
- "Cannot use map-side combining with array keys."
- ]
- },
- "_LEGACY_ERROR_TEMP_3009" : {
- "message" : [
- "HashPartitioner cannot partition array keys."
- ]
- },
- "_LEGACY_ERROR_TEMP_3010" : {
- "message" : [
- "reduceByKeyLocally() does not support array keys"
- ]
- },
"_LEGACY_ERROR_TEMP_3011" : {
"message" : [
"This RDD lacks a SparkContext. It could happen in the following cases:",
diff --git a/core/src/main/scala/org/apache/spark/errors/SparkCoreErrors.scala
b/core/src/main/scala/org/apache/spark/errors/SparkCoreErrors.scala
index f058a7cd3eda..95a1f44ef090 100644
--- a/core/src/main/scala/org/apache/spark/errors/SparkCoreErrors.scala
+++ b/core/src/main/scala/org/apache/spark/errors/SparkCoreErrors.scala
@@ -107,19 +107,25 @@ private[spark] object SparkCoreErrors {
def cannotUseMapSideCombiningWithArrayKeyError(): Throwable = {
new SparkException(
- errorClass = "_LEGACY_ERROR_TEMP_3008", messageParameters = Map.empty,
cause = null
+ errorClass = "UNSUPPORTED_ARRAY_KEY.MAP_SIDE_COMBINE",
+ messageParameters = Map.empty,
+ cause = null
)
}
def hashPartitionerCannotPartitionArrayKeyError(): Throwable = {
new SparkException(
- errorClass = "_LEGACY_ERROR_TEMP_3009", messageParameters = Map.empty,
cause = null
+ errorClass = "UNSUPPORTED_ARRAY_KEY.HASH_PARTITIONER",
+ messageParameters = Map.empty,
+ cause = null
)
}
def reduceByKeyLocallyNotSupportArrayKeysError(): Throwable = {
new SparkException(
- errorClass = "_LEGACY_ERROR_TEMP_3010", messageParameters = Map.empty,
cause = null
+ errorClass = "UNSUPPORTED_ARRAY_KEY.REDUCE_BY_KEY_LOCALLY",
+ messageParameters = Map.empty,
+ cause = null
)
}
diff --git a/core/src/test/scala/org/apache/spark/PartitioningSuite.scala
b/core/src/test/scala/org/apache/spark/PartitioningSuite.scala
index d0423e267baf..1dca5c6c11d5 100644
--- a/core/src/test/scala/org/apache/spark/PartitioningSuite.scala
+++ b/core/src/test/scala/org/apache/spark/PartitioningSuite.scala
@@ -215,24 +215,34 @@ class PartitioningSuite extends SparkFunSuite with
SharedSparkContext with Priva
val arrPairs: RDD[(Array[Int], Int)] =
sc.parallelize(Array(1, 2, 3, 4).toImmutableArraySeq, 2).map(x =>
(Array(x), x))
- def verify(testFun: => Unit): Unit = {
- intercept[SparkException](testFun).getMessage.contains("array")
+ def verify(subCondition: String)(testFun: => Unit): Unit = {
+ checkError(
+ exception = intercept[SparkException](testFun),
+ condition = s"UNSUPPORTED_ARRAY_KEY.$subCondition",
+ sqlState = Some("0A000"))
}
- verify(arrs.distinct())
+ // combineByKeyWithClassTag checks mapSideCombine before it looks at the
partitioner, so the
+ // calls below that leave mapSideCombine at its default report
MAP_SIDE_COMBINE even though
+ // they also end up with a HashPartitioner.
+ verify("MAP_SIDE_COMBINE")(arrs.distinct())
// We can't catch all usages of arrays, since they might occur inside
other collections:
// assert(fails { arrPairs.distinct() })
- verify(arrPairs.partitionBy(new HashPartitioner(2)))
- verify(arrPairs.join(arrPairs))
- verify(arrPairs.leftOuterJoin(arrPairs))
- verify(arrPairs.rightOuterJoin(arrPairs))
- verify(arrPairs.fullOuterJoin(arrPairs))
- verify(arrPairs.groupByKey())
- verify(arrPairs.countByKey())
- verify(arrPairs.countByKeyApprox(1))
- verify(arrPairs.cogroup(arrPairs))
- verify(arrPairs.reduceByKeyLocally(_ + _))
- verify(arrPairs.reduceByKey(_ + _))
+ verify("HASH_PARTITIONER")(arrPairs.partitionBy(new HashPartitioner(2)))
+ verify("HASH_PARTITIONER")(arrPairs.join(arrPairs))
+ verify("HASH_PARTITIONER")(arrPairs.leftOuterJoin(arrPairs))
+ verify("HASH_PARTITIONER")(arrPairs.rightOuterJoin(arrPairs))
+ verify("HASH_PARTITIONER")(arrPairs.fullOuterJoin(arrPairs))
+ verify("HASH_PARTITIONER")(arrPairs.groupByKey())
+ verify("MAP_SIDE_COMBINE")(arrPairs.countByKey())
+ // countByKeyApprox() is rejected by RDD.countByValueApprox, whose own
array check is a
+ // separate condition that is still legacy.
+ checkError(
+ exception = intercept[SparkException](arrPairs.countByKeyApprox(1)),
+ condition = "_LEGACY_ERROR_TEMP_3015")
+ verify("HASH_PARTITIONER")(arrPairs.cogroup(arrPairs))
+ verify("REDUCE_BY_KEY_LOCALLY")(arrPairs.reduceByKeyLocally(_ + _))
+ verify("MAP_SIDE_COMBINE")(arrPairs.reduceByKey(_ + _))
}
test("zero-length partitions should be correctly handled") {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
