Chao Sun created SPARK-36137:
--------------------------------
Summary: HiveShim always fallback to getAllPartitionsOf regardless
of whether directSQL is enabled in remote HMS
Key: SPARK-36137
URL: https://issues.apache.org/jira/browse/SPARK-36137
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.2.0
Reporter: Chao Sun
At the moment {{getPartitionsByFilter}} in Hive shim only fallback to use
{{getAllPartitionsOf}} when {{hive.metastore.try.direct.sql}} is enabled in the
remote HMS. However, in certain cases the remote HMS will fallback to use ORM
to query the underlying RDBMS even if this config is set to true, and Spark
will not be able to recover from the error and will just fail the query.
For instance, we encountered this bug HIVE-21497 in HMS running Hive 3.1.2, and
Spark was not able to pushdown filter for {{date}} column.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
