cxzl25 commented on code in PR #47998:
URL: https://github.com/apache/spark/pull/47998#discussion_r1758392635
##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala:
##########
@@ -415,8 +415,11 @@ private[client] class Shim_v2_0 extends Shim with Logging {
try {
val partitionSchema =
CharVarcharUtils.replaceCharVarcharWithStringInSchema(
catalogTable.partitionSchema)
+ val transformedPredicates = predicates.map(_.transform {
+ case a: AttributeReference =>
a.withName(a.name.toLowerCase(Locale.ROOT))
+ })
val boundPredicate =
ExternalCatalogUtils.generatePartitionPredicateByFilter(
- catalogTable, partitionSchema, predicates)
+ catalogTable, partitionSchema, transformedPredicates)
Review Comment:
Because the catalogTable here is loaded directly from HMS, the partition
fields are in lowercase instead of being restored to uppercase partition
fields, which leads to this problem.
Because the catalog of `conf.metastorePartitionPruning=false` has partition
field names in uppercase, there is no such problem.
https://github.com/apache/spark/blob/319e7cc7d0e7ba9a99f808d51a8d635a6159ce8f/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogUtils.scala#L163-L173
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
