Ryan Blue created SPARK-18086:
---------------------------------
Summary: Regression: Hive variables no longer work in Spark 2.0
Key: SPARK-18086
URL: https://issues.apache.org/jira/browse/SPARK-18086
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.0.0
Reporter: Ryan Blue
The behavior of variables in the SQL shell has changed from 1.6 to 2.0.
Specifically, --hivevar name=value and {{SET hivevar:name=value}} no longer
work. Queries that worked correctly in 1.6 will either fail or produce
unexpected results in 2.0 so I think this is a regression that should be
addressed.
Hive and Spark 1.6 work like this:
1. Command-line args --hiveconf and --hivevar can be used to set session
properties. --hiveconf properties are added to the Hadoop Configuration.
2. {{SET}} adds a Hive Configuration property, {{SET hivevar:<name>=<value>}}
adds a Hive var.
3. Hive vars can be substituted into queries by name, and Configuration
properties can be substituted using {{hiveconf:name}}.
In 2.0, hiveconf, sparkconf, and conf variable prefixes are all removed, then
the value in SQLConf for the rest of the key is returned. SET adds properties
to the session config and (according to [a
comment|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/RuntimeConfig.scala#L28])
the Hadoop configuration "during I/O".
{code:title=Hive and Spark 1.6.1 behavior}
[user@host:~]: spark-sql --hiveconf test.conf=1 --hivevar test.var=2
spark-sql> select "${hiveconf:test.conf}";
1
spark-sql> select "${test.conf}";
${test.conf}
spark-sql> select "${hivevar:test.var}";
2
spark-sql> select "${test.var}";
2
spark-sql> set test.set=3;
SET test.set=3
spark-sql> select "${test.set}"
"${test.set}"
spark-sql> select "${hivevar:test.set}"
"${hivevar:test.set}"
spark-sql> select "${hiveconf:test.set}"
3
spark-sql> set hivevar:test.setvar=4;
SET hivevar:test.setvar=4
spark-sql> select "${hivevar:test.setvar}";
4
spark-sql> select "${test.setvar}";
4
{code}
{code:title=Spark 2.0.0 behavior}
[user@host:~]: spark-sql --hiveconf test.conf=1 --hivevar test.var=2
spark-sql> select "${hiveconf:test.conf}";
1
spark-sql> select "${test.conf}";
1
spark-sql> select "${hivevar:test.var}";
${hivevar:test.var}
spark-sql> select "${test.var}";
${test.var}
spark-sql> set test.set=3;
test.set 3
spark-sql> select "${test.set}";
3
spark-sql> set hivevar:test.setvar=4;
hivevar:test.setvar 4
spark-sql> select "${hivevar:test.setvar}";
4
spark-sql> select "${test.setvar}";
${test.setvar}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
