MaxGekk commented on code in PR #41553:
URL: https://github.com/apache/spark/pull/41553#discussion_r1234980739
##########
core/src/main/resources/error/error-classes.json:
##########
@@ -2388,6 +2403,11 @@
"<property> is a reserved namespace property, <msg>."
]
},
+ "SET_OPERATION_ON_MAP_TYPE_UNSUPPORTED" : {
+ "message" : [
+ "Cannot have map type columns in DataFrame which calls set
operations(intersect, except, etc.), but the type of column <colName> is
<dataType>."
Review Comment:
Could you follow the style of `toSQLType` and `toSQLStmt`
```suggestion
"Cannot have MAP type columns in DataFrame which calls set
operations (INTERSECT, EXCEPT, etc.), but the type of column <colName> is
<dataType>."
```
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/UpdateTableSuiteBase.scala:
##########
@@ -538,7 +538,7 @@ abstract class UpdateTableSuiteBase extends
RowLevelOperationSuiteBase {
val e = intercept[AnalysisException] {
sql(s"UPDATE $tableNameAsString SET dep = 'invalid' WHERE id <= 1 AND
rand() > 0.5")
}
- assert(e.message.contains("nondeterministic expressions are only allowed"))
+ assert(e.message.contains("The operator expects a deterministic
expression"))
Review Comment:
Could you use `checkError`
##########
core/src/main/resources/error/error-classes.json:
##########
@@ -2388,6 +2403,11 @@
"<property> is a reserved namespace property, <msg>."
]
},
+ "SET_OPERATION_ON_MAP_TYPE_UNSUPPORTED" : {
Review Comment:
You can omit `_UNSUPPORTED` in the sub-class of `UNSUPPORTED_FEATURE`.
##########
sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala:
##########
@@ -352,20 +352,40 @@ class DataFrameSetOperationsSuite extends QueryTest with
SharedSparkSession {
test("SPARK-19893: cannot run set operations with map type") {
val df = spark.range(1).select(map(lit("key"), $"id").as("m"))
- val e = intercept[AnalysisException](df.intersect(df))
- assert(e.message.contains(
- "Cannot have map type columns in DataFrame which calls set operations"))
- val e2 = intercept[AnalysisException](df.except(df))
- assert(e2.message.contains(
- "Cannot have map type columns in DataFrame which calls set operations"))
- val e3 = intercept[AnalysisException](df.distinct())
- assert(e3.message.contains(
- "Cannot have map type columns in DataFrame which calls set operations"))
+ checkError(
+ exception = intercept[AnalysisException](df.intersect(df)),
+ errorClass = "UNSUPPORTED_FEATURE.SET_OPERATION_ON_MAP_TYPE_UNSUPPORTED",
+ parameters = Map(
+ "colName" -> "\"m\"",
Review Comment:
Shouldn't `m` be considered as an id and quoted by backticks instead of ""?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
