Hu Fuwang created SPARK-30525:
---------------------------------
Summary: HiveTableScanExec do not need to prune partitions again
after pushing down to hive metastore
Key: SPARK-30525
URL: https://issues.apache.org/jira/browse/SPARK-30525
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.1.0
Reporter: Hu Fuwang
In HiveTableScanExec, it will push down to hive metastore for partition pruning
if
spark.sql.hive.metastorePartitionPruning is true, and then it will prune the
returned partitions again using partition filters, because some predicates, eg.
"b like 'xyz'", are not supported in hive metastore. But now this problem is
already fixed in HiveExternalCatalog.listPartitionsByFilter, the
HiveExternalCatalog.listPartitionsByFilter can return exactly what we want now.
So it is not necessary any more to double prune in HiveTableScanExec.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
