[ https://issues.apache.org/jira/browse/SPARK-36137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381669#comment-17381669 ]
Apache Spark commented on SPARK-36137: -------------------------------------- User 'sunchao' has created a pull request for this issue: https://github.com/apache/spark/pull/33382 > HiveShim always fallback to getAllPartitionsOf regardless of whether > directSQL is enabled in remote HMS > ------------------------------------------------------------------------------------------------------- > > Key: SPARK-36137 > URL: https://issues.apache.org/jira/browse/SPARK-36137 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.2.0 > Reporter: Chao Sun > Priority: Major > > At the moment {{getPartitionsByFilter}} in Hive shim only fallback to use > {{getAllPartitionsOf}} when {{hive.metastore.try.direct.sql}} is enabled in > the remote HMS. However, in certain cases the remote HMS will fallback to use > ORM (which only support string type for partition columns) to query the > underlying RDBMS even if this config is set to true, and Spark will not be > able to recover from the error and will just fail the query. > For instance, we encountered this bug HIVE-21497 in HMS running Hive 3.1.2, > and Spark was not able to pushdown filter for {{date}} column. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org