Herman van Hovell created SPARK-22431:
-----------------------------------------
Summary: Creating Permanent view with illegal type
Key: SPARK-22431
URL: https://issues.apache.org/jira/browse/SPARK-22431
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.2.0
Reporter: Herman van Hovell
Priority: Major
It is possible in Spark SQL to create a permanent view that uses an nested
field with an illegal name.
For example if we create the following view:
{noformat}
create view x as select struct('a' as `$q`, 1 as b) q
{noformat}
A simple select fails with the following exception:
{noformat}
select * from x;
org.apache.spark.SparkException: Cannot recognize hive type string:
struct<$q:string,b:int>
at
org.apache.spark.sql.hive.client.HiveClientImpl$.fromHiveColumn(HiveClientImpl.scala:812)
at
org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1$$anonfun$apply$11$$anonfun$7.apply(HiveClientImpl.scala:378)
at
org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1$$anonfun$apply$11$$anonfun$7.apply(HiveClientImpl.scala:378)
...
{noformat}
Dropping the view isn't possible either:
{noformat}
drop view x;
org.apache.spark.SparkException: Cannot recognize hive type string:
struct<$q:string,b:int>
at
org.apache.spark.sql.hive.client.HiveClientImpl$.fromHiveColumn(HiveClientImpl.scala:812)
at
org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1$$anonfun$apply$11$$anonfun$7.apply(HiveClientImpl.scala:378)
at
org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1$$anonfun$apply$11$$anonfun$7.apply(HiveClientImpl.scala:378)
...
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
