This is an automated email from the ASF dual-hosted git repository.
LuciferYang pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 427fb7db38d7 [SPARK-59040][SQL] Assign a name to the error condition
`_LEGACY_ERROR_TEMP_1166,1167`
427fb7db38d7 is described below
commit 427fb7db38d700434503b6db0679ca4f918a722f
Author: YangJie <[email protected]>
AuthorDate: Tue Sep 1 11:32:11 2026 +0800
[SPARK-59040][SQL] Assign a name to the error condition
`_LEGACY_ERROR_TEMP_1166,1167`
### What changes were proposed in this pull request?
This PR assigns proper names to two placeholder error conditions, both with
SQLSTATE `42713`:
| legacy condition | new condition |
|---|---|
| `_LEGACY_ERROR_TEMP_1166` | `BUCKET_COLUMN_IN_PARTITION_COLUMNS` |
| `_LEGACY_ERROR_TEMP_1167` | `BUCKET_SORT_COLUMN_IN_PARTITION_COLUMNS` |
The first message is unchanged. The second now says "Bucket sort column"
where it said "Bucket sorting column", so that the sentence, the condition name
and the parameter key all use the same term. Two things about how both render
also change: the column names go through `toSQLId(Seq(...))` instead of the
legacy single quotes, and the three message parameters are renamed from
`bucketCol`, `sortCol` and `normalizedPartCols` to `bucketColumn`, `sortColumn`
and `partitionColumns`. Both b [...]
### Why are the changes needed?
This continues the work of replacing `_LEGACY_ERROR_TEMP_*` placeholders
with real condition names. Both of these are reachable from user code:
`normalizeCatalogTable` runs for `CREATE TABLE`, CTAS and
`df.write.partitionBy(...).bucketBy(...).saveAsTable(...)`, and either error
fires when a column is named in both the partition spec and the bucket spec. A
proper name is the right outcome here, not an internal error.
The names follow how this file already splits the vocabulary: `BUCKET_*`
when it means the object, `BUCKETING` when it means the feature or the clause.
Hence `BUCKET_COLUMN` and `BUCKET_SORT_COLUMN`, alongside
`INVALID_BUCKET_COLUMN_DATA_TYPE` and `CANNOT_ALTER_COLLATION_BUCKET_COLUMN` on
one side and `SORT_BY_WITHOUT_BUCKETING` and `SPECIFY_BUCKETING_IS_NOT_ALLOWED`
on the other. `BucketSpec` calls the fields `bucketColumnNames` and
`sortColumnNames`. Both stay top-level rather than [...]
`42713` reads "A duplicate object was detected in a list or is the same as
an existing object", and the second half of that is what happens here: a column
named in `CLUSTERED BY` is the same as one already named in `PARTITIONED BY`.
The closest match in the repo is
`STATIC_PARTITION_COLUMN_IN_INSERT_COLUMN_LIST`, "Static partition column
`<staticName>` is also specified in the column list", which is both the same
shape and the precedent for the `X_IN_Y` name. It also leaves a readable [...]
On the quoting, `QueryErrorsBase`'s scaladoc asks for identifiers to be
wrapped in backticks via `toSQLId()`, which `QueryCompilationErrors` already
does throughout. The `Seq` overload is the right one here: these names arrive
already resolved against the table schema, so each is a single part, whereas
`toSQLId(String)` runs them through `AttributeNameParser.parseAttributeName`
first and would render a column literally named `x.y` as `` `x`.`y` ``, or fail
with `INVALID_ATTRIBUTE_NAME [...]
The parameter names are worth settling now for a similar reason of timing:
they appear in the generated documentation and in `getMessageParameters()`, and
once the conditions ship the keys cannot move without breaking anything that
matches on them. `normalizedPartCols` named an internal normalization step that
means nothing to a reader. The three now read as the singular and plural of one
noun, which is also how this file spells them: `...Columns` is the prevailing
name for a list of [...]
### Does this PR introduce _any_ user-facing change?
Yes, in the rendered message. A named condition gains the `[CONDITION] `
prefix and the ` SQLSTATE: ...` suffix that
`SparkThrowableHelper.formatErrorMessage` suppresses for
`_LEGACY_ERROR_`-prefixed names, and the column names are now backticked:
```
-Bucketing column 'j' should not be part of partition columns 'i, j'.
+[BUCKET_COLUMN_IN_PARTITION_COLUMNS] Bucketing column `j` should not be
part of partition columns `i`, `j`. SQLSTATE: 42713
-Bucket sorting column 'i' should not be part of partition columns 'i, j'.
+[BUCKET_SORT_COLUMN_IN_PARTITION_COLUMNS] Bucket sort column `i` should
not be part of partition columns `i`, `j`. SQLSTATE: 42713
```
Code that reads `getMessageParameters()` sees `bucketColumn`, `sortColumn`
and `partitionColumns` where it saw `bucketCol`, `sortCol` and
`normalizedPartCols`. The two conditions also start showing up in the generated
error conditions documentation, which skips `_LEGACY_ERROR_*` entries.
### How was this patch tested?
`BucketedWriteSuite`'s existing assertions for both conditions now check
the new names, the backticked parameter values and `sqlState = "42713"`. A new
test in the same suite writes a table whose columns are named `x.y` and
``z`w``, and pins them as `` `x.y` `` and `` `z``w` ``; against the
`toSQLId(String)` overload that test instead fails with
`INVALID_ATTRIBUTE_NAME_SYNTAX`, which is what a user would have seen. Ran
`core/testOnly org.apache.spark.SparkThrowableSuite` and `sql/test [...]
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)
Closes #58333 from LuciferYang/assign-name-legacy-1166-1167.
Authored-by: YangJie <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
(cherry picked from commit c89c0b5974fde219cd85c2887b47912dcfa88f83)
Signed-off-by: yangjie01 <[email protected]>
---
.../src/main/resources/error/error-conditions.json | 22 +++++++++++---------
.../spark/sql/errors/QueryCompilationErrors.scala | 17 ++++++++-------
.../spark/sql/sources/BucketedWriteSuite.scala | 24 ++++++++++++++++++----
3 files changed, 42 insertions(+), 21 deletions(-)
diff --git a/common/utils/src/main/resources/error/error-conditions.json
b/common/utils/src/main/resources/error/error-conditions.json
index 0b1dbb183978..62ff7314a8d7 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -495,6 +495,18 @@
],
"sqlState" : "22001"
},
+ "BUCKET_COLUMN_IN_PARTITION_COLUMNS" : {
+ "message" : [
+ "Bucketing column <bucketColumn> should not be part of partition columns
<partitionColumns>."
+ ],
+ "sqlState" : "42713"
+ },
+ "BUCKET_SORT_COLUMN_IN_PARTITION_COLUMNS" : {
+ "message" : [
+ "Bucket sort column <sortColumn> should not be part of partition columns
<partitionColumns>."
+ ],
+ "sqlState" : "42713"
+ },
"CALL_ON_STREAMING_DATASET_UNSUPPORTED" : {
"message" : [
"The method <methodName> can not be called on streaming
Dataset/DataFrame."
@@ -10397,16 +10409,6 @@
"It is not allowed to specify partitioning when the table schema is not
defined."
]
},
- "_LEGACY_ERROR_TEMP_1166" : {
- "message" : [
- "Bucketing column '<bucketCol>' should not be part of partition columns
'<normalizedPartCols>'."
- ]
- },
- "_LEGACY_ERROR_TEMP_1167" : {
- "message" : [
- "Bucket sorting column '<sortCol>' should not be part of partition
columns '<normalizedPartCols>'."
- ]
- },
"_LEGACY_ERROR_TEMP_1169" : {
"message" : [
"Requested partitioning does not match the table <tableName>:",
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
index a662eee4ea41..3b5d7accd65b 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
@@ -2229,20 +2229,23 @@ private[sql] object QueryCompilationErrors extends
QueryErrorsBase with Compilat
def bucketingColumnCannotBePartOfPartitionColumnsError(
bucketCol: String, normalizedPartCols: Seq[String]): Throwable = {
+ // These names are already resolved against the table schema, so each is
quoted as a single
+ // part. The `String` overload of `toSQLId` would parse them instead.
new AnalysisException(
- errorClass = "_LEGACY_ERROR_TEMP_1166",
+ errorClass = "BUCKET_COLUMN_IN_PARTITION_COLUMNS",
messageParameters = Map(
- "bucketCol" -> bucketCol,
- "normalizedPartCols" -> normalizedPartCols.mkString(", ")))
+ "bucketColumn" -> toSQLId(Seq(bucketCol)),
+ "partitionColumns" -> normalizedPartCols.map(c =>
toSQLId(Seq(c))).mkString(", ")))
}
def bucketSortingColumnCannotBePartOfPartitionColumnsError(
- sortCol: String, normalizedPartCols: Seq[String]): Throwable = {
+ sortCol: String, normalizedPartCols: Seq[String]): Throwable = {
+ // Quoted as single parts, as in
bucketingColumnCannotBePartOfPartitionColumnsError above.
new AnalysisException(
- errorClass = "_LEGACY_ERROR_TEMP_1167",
+ errorClass = "BUCKET_SORT_COLUMN_IN_PARTITION_COLUMNS",
messageParameters = Map(
- "sortCol" -> sortCol,
- "normalizedPartCols" -> normalizedPartCols.mkString(", ")))
+ "sortColumn" -> toSQLId(Seq(sortCol)),
+ "partitionColumns" -> normalizedPartCols.map(c =>
toSQLId(Seq(c))).mkString(", ")))
}
def invalidBucketColumnDataTypeError(dataType: DataType): Throwable = {
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedWriteSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedWriteSuite.scala
index 1423df2f3550..ad485033e672 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedWriteSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedWriteSuite.scala
@@ -252,8 +252,9 @@ abstract class BucketedWriteSuite extends QueryTest {
.bucketBy(8, "j", "k")
.sortBy("k")
.saveAsTable("bucketed_table")),
- condition = "_LEGACY_ERROR_TEMP_1166",
- parameters = Map("bucketCol" -> "j", "normalizedPartCols" -> "i, j"))
+ condition = "BUCKET_COLUMN_IN_PARTITION_COLUMNS",
+ sqlState = "42713",
+ parameters = Map("bucketColumn" -> "`j`", "partitionColumns" -> "`i`,
`j`"))
checkError(
exception = intercept[AnalysisException](df.write
@@ -261,8 +262,23 @@ abstract class BucketedWriteSuite extends QueryTest {
.bucketBy(8, "k")
.sortBy("i")
.saveAsTable("bucketed_table")),
- condition = "_LEGACY_ERROR_TEMP_1167",
- parameters = Map("sortCol" -> "i", "normalizedPartCols" -> "i, j"))
+ condition = "BUCKET_SORT_COLUMN_IN_PARTITION_COLUMNS",
+ sqlState = "42713",
+ parameters = Map("sortColumn" -> "`i`", "partitionColumns" -> "`i`,
`j`"))
+ }
+
+ test("SPARK-59040: overlapping column names are quoted as single
identifiers") {
+ // A name is already resolved here, so it must not be parsed: "x.y" is one
column rather than
+ // a qualified reference, and a name holding a backtick has it escaped
rather than rejected.
+ val df = Seq((1, "a", 2)).toDF("x.y", "z`w", "k")
+ checkError(
+ exception = intercept[AnalysisException](df.write
+ .partitionBy("x.y", "z`w")
+ .bucketBy(8, "x.y")
+ .saveAsTable("bucketed_table")),
+ condition = "BUCKET_COLUMN_IN_PARTITION_COLUMNS",
+ sqlState = "42713",
+ parameters = Map("bucketColumn" -> "`x.y`", "partitionColumns" ->
"`x.y`, `z``w`"))
}
test("write bucketed data without partitionBy") {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
