fuwhu commented on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to hive metastore URL: https://github.com/apache/spark/pull/27232#issuecomment-575660407 > emmmm, prune partition again is because when call `listPartitionsByFilter` kit can't convert all spark filter expression to hive metastore filter condition. and will get more partition then exactly wanted. then prune again will spark's method to keep smallest partitions. > > If we can't promise all these case was fixed in `listPartitionsByFilter`, we may still need this. and it's cost is negligible HiveExternalCatalog.listPartitionsByFilter will call HiveClient.getPartitionsByFilter to push down to hive metastore for partition pruning, which may not convert all spark filters to hive filters. But now it already call ExternalCatalogUtils.prunePartitionsByFilter to prune the results returned by HiveClient.getPartitionsByFilter again in HiveExternalCatalog.listPartitionsByFilter. So it is not necessary any more to prune again in HiveTableScanExec.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
