[jira] [Commented] (SPARK-25561) HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql
[ https://issues.apache.org/jira/browse/SPARK-25561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635945#comment-16635945 ] Karthik Manamcheri commented on SPARK-25561: Created PR https://github.com/apache/spark/pull/22614 > HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql > -- > > Key: SPARK-25561 > URL: https://issues.apache.org/jira/browse/SPARK-25561 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 >Reporter: Karthik Manamcheri >Priority: Major > > In HiveShim.scala, the current behavior is that if > hive.metastore.try.direct.sql is enabled, we expect the getPartitionsByFilter > call to succeed. If it fails, we'll throw a RuntimeException. > However, this might not always be the case. Hive's direct SQL functionality > is best-attempt. Meaning, it will fall back to ORM if direct sql fails. Spark > should handle that exception correctly if Hive falls back to ORM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25561) HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql
[ https://issues.apache.org/jira/browse/SPARK-25561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634332#comment-16634332 ] Karthik Manamcheri commented on SPARK-25561: I am working on a patch for this and will post a PR as soon as possible. > HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql > -- > > Key: SPARK-25561 > URL: https://issues.apache.org/jira/browse/SPARK-25561 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 >Reporter: Karthik Manamcheri >Priority: Major > > In HiveShim.scala, the current behavior is that if > hive.metastore.try.direct.sql is enabled, we expect the getPartitionsByFilter > call to succeed. If it fails, we'll throw a RuntimeException. > However, this might not always be the case. Hive's direct SQL functionality > is best-attempt. Meaning, it will fall back to ORM if direct sql fails. Spark > should handle that exception correctly if Hive falls back to ORM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25561) HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql
[ https://issues.apache.org/jira/browse/SPARK-25561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632483#comment-16632483 ] Karthik Manamcheri commented on SPARK-25561: [~michael] thanks for the prompt reply. This is hard to test because the problem happens only in the case when HMS goes into fallback ORM mode. For this to happen, we need to have the direct SQL query fail in HMS. There are no consistent bugs (that I know of) which can be used to test this in a deterministic fashion. I was able to run into this running Hive 1.1.0. However, as I understand HMS behavior of falling back to ORM has been the same in Hive from the beginning. Not sure. > HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql > -- > > Key: SPARK-25561 > URL: https://issues.apache.org/jira/browse/SPARK-25561 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 >Reporter: Karthik Manamcheri >Priority: Major > > In HiveShim.scala, the current behavior is that if > hive.metastore.try.direct.sql is enabled, we expect the getPartitionsByFilter > call to succeed. If it fails, we'll throw a RuntimeException. > However, this might not always be the case. Hive's direct SQL functionality > is best-attempt. Meaning, it will fall back to ORM if direct sql fails. Spark > should handle that exception correctly if Hive falls back to ORM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25561) HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql
[ https://issues.apache.org/jira/browse/SPARK-25561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631193#comment-16631193 ] Karthik Manamcheri commented on SPARK-25561: The root cause was from SPARK-17992 ping [~michael] what are your thoughts on this? > HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql > -- > > Key: SPARK-25561 > URL: https://issues.apache.org/jira/browse/SPARK-25561 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 >Reporter: Karthik Manamcheri >Priority: Major > > In HiveShim.scala, the current behavior is that if > hive.metastore.try.direct.sql is enabled, we expect the getPartitionsByFilter > call to succeed. If it fails, we'll throw a RuntimeException. > However, this might not always be the case. Hive's direct SQL functionality > is best-attempt. Meaning, it will fall back to ORM if direct sql fails. Spark > should handle that exception correctly if Hive falls back to ORM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-25561) HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql
Karthik Manamcheri created SPARK-25561: -- Summary: HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql Key: SPARK-25561 URL: https://issues.apache.org/jira/browse/SPARK-25561 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.1.0 Reporter: Karthik Manamcheri In HiveShim.scala, the current behavior is that if hive.metastore.try.direct.sql is enabled, we expect the getPartitionsByFilter call to succeed. If it fails, we'll throw a RuntimeException. However, this might not always be the case. Hive's direct SQL functionality is best-attempt. Meaning, it will fall back to ORM if direct sql fails. Spark should handle that exception correctly if Hive falls back to ORM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
