peter-toth commented on code in PR #36027:
URL: https://github.com/apache/spark/pull/36027#discussion_r975119000
##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:
##########
@@ -1026,7 +1026,14 @@ private[hive] object HiveClientImpl extends Logging {
} else {
CharVarcharUtils.getRawTypeString(c.metadata).getOrElse(c.dataType.catalogString)
}
- new FieldSchema(c.name, typeString, c.getComment().orNull)
+ val name = if (lowerCase) {
+ // scalastyle:off caselocale
+ c.name.toLowerCase
+ // scalastyle:on caselocale
+ } else {
+ c.name
Review Comment:
In the last commit
(https://github.com/apache/spark/pull/36027/commits/d7730e512497a09cf9a8564f4f47dd3447e8f6c4)
I lowercased the data column names (and bucketspec columns) but kept the
partitioning columns. This is because Hive seems to be case preserving with
partitioning columns too (despite partition paths (`.../p=v/...`) are
lowercased) and at many places in Spark we use that to match partitioning
columns from Hive to `CatalogTable.partitionColumnNames`.
But even with this commit, when we lowercase data columns only, we have test
failures like `Create Hive table as select should output correct schema`
(https://github.com/apache/spark/blob/master/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala#L942-L964)
as we expect the schema of the parquet file to match the create table
statement.
I feel we should stay on the safe side and simply just allign the bucketspec
in `toHivetable` to the columns as the original commit
(https://github.com/apache/spark/pull/36027/commits/f68ac14d1ff53c6b3104da5fb952495a97d5a4d1)
did.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
