[
https://issues.apache.org/jira/browse/SPARK-36137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated SPARK-36137:
-----------------------------
Description:
At the moment {{getPartitionsByFilter}} in Hive shim only fallback to use
{{getAllPartitionsOf}} when {{hive.metastore.try.direct.sql}} is enabled in the
remote HMS. However, in certain cases the remote HMS will fallback to use ORM
(which only support string type for partition columns) to query the underlying
RDBMS even if this config is set to true, and Spark will not be able to recover
from the error and will just fail the query.
For instance, we encountered this bug HIVE-21497 in HMS running Hive 3.1.2, and
Spark was not able to pushdown filter for {{date}} column.
was:
At the moment {{getPartitionsByFilter}} in Hive shim only fallback to use
{{getAllPartitionsOf}} when {{hive.metastore.try.direct.sql}} is enabled in the
remote HMS. However, in certain cases the remote HMS will fallback to use ORM
to query the underlying RDBMS even if this config is set to true, and Spark
will not be able to recover from the error and will just fail the query.
For instance, we encountered this bug HIVE-21497 in HMS running Hive 3.1.2, and
Spark was not able to pushdown filter for {{date}} column.
> HiveShim always fallback to getAllPartitionsOf regardless of whether
> directSQL is enabled in remote HMS
> -------------------------------------------------------------------------------------------------------
>
> Key: SPARK-36137
> URL: https://issues.apache.org/jira/browse/SPARK-36137
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.2.0
> Reporter: Chao Sun
> Priority: Major
>
> At the moment {{getPartitionsByFilter}} in Hive shim only fallback to use
> {{getAllPartitionsOf}} when {{hive.metastore.try.direct.sql}} is enabled in
> the remote HMS. However, in certain cases the remote HMS will fallback to use
> ORM (which only support string type for partition columns) to query the
> underlying RDBMS even if this config is set to true, and Spark will not be
> able to recover from the error and will just fail the query.
> For instance, we encountered this bug HIVE-21497 in HMS running Hive 3.1.2,
> and Spark was not able to pushdown filter for {{date}} column.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
