gengliangwang commented on code in PR #36077:
URL: https://github.com/apache/spark/pull/36077#discussion_r845248297
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala:
##########
@@ -239,12 +268,33 @@ case class ResolveDefaultColumns(
val lookup = catalog.lookupRelation(tableName)
lookup match {
case SubqueryAlias(_, r: UnresolvedCatalogRelation) =>
- Some(StructType(r.tableMeta.schema.fields.dropRight(
- enclosingInsert.get.partitionSpec.size)))
- case _ => None
+ StructType(r.tableMeta.schema.fields.dropRight(
+ enclosingInsert.get.partitionSpec.size))
+ case _ => return None
}
} catch {
- case _: NoSuchTableException => None
+ case _: AnalysisException => return None
+ }
+ // Rearrange the columns in the result schema to match the order of the
explicit column list,
+ // if any.
+ val userSpecifiedCols: Seq[String] = enclosingInsert.get.userSpecifiedCols
+ if (userSpecifiedCols.isEmpty) {
+ return Some(schema)
}
+ val colNamesToFields: Map[String, StructField] =
+ schema.fields.map {
+ field: StructField => field.name -> field
+ }.toMap
+ val userSpecifiedFields: Seq[StructField] =
+ userSpecifiedCols.map {
+ name: String => colNamesToFields.getOrElse(name, return None)
Review Comment:
There is a config `spark.sql.caseSensitive` and it is false by default.
So we need to make sure that this rule is executed after the rule Analysis
rule `ResolveUserSpecifiedColumns`.
Probably we can:
1. set a TreeNodeTag `UserSpecifiedColumnsResolved` in
`ResolveUserSpecifiedColumns`
2. If there is user specified column list, only execute this rule when there
is a TreeNodeTag `UserSpecifiedColumnsResolved`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
