Github user mallman commented on a diff in the pull request:
    --- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
    @@ -616,6 +617,44 @@ private[spark] class HiveExternalCatalog(conf: 
SparkConf, hadoopConf: Configurat
         client.getPartition(db, table, spec)
    +  override def listPartitionsByFilter(
    +      db: String,
    +      table: String,
    +      predicates: Seq[Expression]): Seq[CatalogTablePartition] = 
withClient {
    +    val catalogTable = client.getTable(db, table)
    +    val partitionColumnNames = catalogTable.partitionColumnNames.toSet
    +    val nonPartitionPruningPredicates = predicates.filterNot {
    +    }
    +    if (nonPartitionPruningPredicates.nonEmpty) {
    +        sys.error("Expected only partition pruning predicates: " +
    +          predicates.reduceLeft(And))
    +    }
    +    val partitionSchema = catalogTable.partitionSchema
    +    if (predicates.nonEmpty) {
    +      val clientPrunedPartitions =
    +        client.getPartitionsByFilter(catalogTable, predicates)
    +      val boundPredicate =
    +        InterpretedPredicate.create(predicates.reduce(And).transform {
    +          case att: AttributeReference =>
    +            val index = partitionSchema.indexWhere( ==
    --- End diff --
    I tested this with unit tests from two test suites on two branches. The 
first test suite was `SQLQuerySuite` from the Hive codebase, specifically the 
test "SPARK-10562: partition by column with mixed case name". The second test 
suite was (a modified) `ParquetMetastoreSuite`. I modified the name of the 
partition column in the partitioned tables in the latter suite from `p` to 
`pQ`. The two branches on which I tested were this PR and commit 8d33e1e from 
the master branch.
    The first test suite passed on both branches. I guess that's to be expected 
since our Jenkins bot has been reporting it as passed.
    The second suite failed (as modified) on both branches. In both branches, 
Spark SQL failed to find the partitions on-disk. This makes me wonder:
    1. Is this a known/accepted limitation?
    1. If unknown, is this an acceptable limitation or a bug to be fixed?
    The best I found regarding support for mixed-case partition columns was in Unlike in the first test 
(which uses the `saveAsTable` method), the tables in `ParquetMetastoreSuite` 
are built with SQL DDL.

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to