dongjoon-hyun commented on a change in pull request #25234:
[SPARK-28054][SQL][followup] move the bug fix closer to where causes the issue
URL: https://github.com/apache/spark/pull/25234#discussion_r306388453
##########
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala
##########
@@ -186,10 +188,15 @@ case class InsertIntoHiveTable(
}
val partitionAttributes =
partitionColumnNames.takeRight(numDynamicPartitions).map { name =>
- query.resolve(name :: Nil,
sparkSession.sessionState.analyzer.resolver).getOrElse {
+ val attr = query.resolve(name :: Nil,
sparkSession.sessionState.analyzer.resolver).getOrElse {
throw new AnalysisException(
s"Unable to resolve $name given
[${query.output.map(_.name).mkString(", ")}]")
}.asInstanceOf[Attribute]
+ // SPARK-28054: Hive metastore is not case preserving and keeps
partition columns
+ // with lower cased names. Hive will validate the column names in the
partition directories
+ // during `loadDynamicPartitions`. Spark needs to write partition
directories with lower-cased
+ // column names in order to make `loadDynamicPartitions` work.
+ attr.withName(name.toLowerCase(Locale.ROOT))
Review comment:
This move looks nicer and also cleaner due to the removal of `scalastyle:off
caselocale`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
