Repository: spark Updated Branches: refs/heads/branch-2.1 35011608f -> 523071f3f
[SPARK-18681][SQL] Fix filtering to compatible with partition keys of type int ## What changes were proposed in this pull request? Cloudera put `/var/run/cloudera-scm-agent/process/15000-hive-HIVEMETASTORE/hive-site.xml` as the configuration file for the Hive Metastore Server, where `hive.metastore.try.direct.sql=false`. But Spark isn't reading this configuration file and get default value `hive.metastore.try.direct.sql=true`. As mallman said, we should use `getMetaConf` method to obtain the original configuration from Hive Metastore Server. I have tested this method few times and the return value is always consistent with Hive Metastore Server. ## How was this patch tested? The existing tests. Author: Yuming Wang <[email protected]> Closes #16122 from wangyum/SPARK-18681. (cherry picked from commit 90abfd15f4b3f612a7b0ff65f03bf319c78a0243) Signed-off-by: Herman van Hovell <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/523071f3 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/523071f3 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/523071f3 Branch: refs/heads/branch-2.1 Commit: 523071f3fae72909b64c7f405868bbc85f5c3cde Parents: 3501160 Author: Yuming Wang <[email protected]> Authored: Mon Dec 12 23:38:36 2016 +0100 Committer: Herman van Hovell <[email protected]> Committed: Mon Dec 12 23:39:13 2016 +0100 ---------------------------------------------------------------------- .../scala/org/apache/spark/sql/hive/client/HiveShim.scala | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/523071f3/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala ---------------------------------------------------------------------- diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala index e561706..87f58e5 100644 --- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala +++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala @@ -590,8 +590,11 @@ private[client] class Shim_v0_13 extends Shim_v0_12 { } else { logDebug(s"Hive metastore filter is '$filter'.") val tryDirectSqlConfVar = HiveConf.ConfVars.METASTORE_TRY_DIRECT_SQL - val tryDirectSql = - hive.getConf.getBoolean(tryDirectSqlConfVar.varname, tryDirectSqlConfVar.defaultBoolVal) + // We should get this config value from the metaStore. otherwise hit SPARK-18681. + // To be compatible with hive-0.12 and hive-0.13, In the future we can achieve this by: + // val tryDirectSql = hive.getMetaConf(tryDirectSqlConfVar.varname).toBoolean + val tryDirectSql = hive.getMSC.getConfigValue(tryDirectSqlConfVar.varname, + tryDirectSqlConfVar.defaultBoolVal.toString).toBoolean try { // Hive may throw an exception when calling this method in some circumstances, such as // when filtering on a non-string partition column when the hive config key --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
