[
https://issues.apache.org/jira/browse/SPARK-11398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhenhua Wang updated SPARK-11398:
---------------------------------
Summary: unnecessary def dialectClassName in HiveContext, and misleading
dialect conf at the start of spark-sql (was: misleading dialect conf at the
start of spark-sql)
> unnecessary def dialectClassName in HiveContext, and misleading dialect conf
> at the start of spark-sql
> ------------------------------------------------------------------------------------------------------
>
> Key: SPARK-11398
> URL: https://issues.apache.org/jira/browse/SPARK-11398
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Zhenhua Wang
> Priority: Minor
>
> 1. def dialectClassName in HiveContext is unnecessary.
> In HiveContext, if conf.dialect == "hiveql", getSQLDialect() will return new
> HiveQLDialect(this);
> else it will use super.getSQLDialect(). Then in super.getSQLDialect(), it
> calls dialectClassName, which is overriden in HiveContext and still return
> super.dialectClassName.
> So we'll never reach the code "classOf[HiveQLDialect].getCanonicalName" of
> def dialectClassName in HiveContext.
> 2. When we start bin/spark-sql, the default context is HiveContext, and the
> corresponding dialect is hiveql.
> However, if we type "set spark.sql.dialect;", the result is "sql", which is
> inconsistent with the actual dialect and is misleading. For example, we can
> use sql like "create table" which is only allowed in hiveql, but this dialect
> conf shows it's "sql".
> Although this problem will not cause any execution error, it's misleading to
> spark sql users. Therefore I think we should fix it.
> In this pr, instead of overriding def dialect in conf of HiveContext, I set
> the SQLConf.DIALECT directly as "hiveql", such that result of "set
> spark.sql.dialect;" will be "hiveql", not "sql". After the change, we can
> still use "sql" as the dialect in HiveContext through "set
> spark.sql.dialect=sql". Then the conf.dialect in HiveContext will become sql.
> Because in SQLConf, def dialect = getConf(), and now the dialect in
> "settings" becomes "sql".
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
