Sujith Chacko created SPARK-27617:
-------------------------------------
Summary: Not able to specify LOCATION for internal table
Key: SPARK-27617
URL: https://issues.apache.org/jira/browse/SPARK-27617
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 2.4.0, 2.3.0, 2.2.0
Reporter: Sujith Chacko
In spark whenever user specifies location uri in create table without external
keyword the table is treated as external table .
Because of this behavior following following problems has been observed
a) user will not able to set an external location for a managed table.
b) compatibility issue with hive impala where the system allow managed table
to specify location uri if user created table without 'EXTERNAL' keyword.
scala> spark.sql("""CREATE TABLE IF NOT EXISTS ext2 (name STRING) LOCATION
'D:/spark-2.4.1-bin-hadoop2.7/bin/spark-warehouse/abc_orc13'""");
-chgrp: 'HTIPL-23270\None' does not match expected pattern for group
Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...
res15: org.apache.spark.sql.DataFrame = []
scala> spark.sql("desc formatted ext2").show(false)
+----------------------------+----------------------------------------------------------------+-------+
|col_name |data_type |comment|
+----------------------------+----------------------------------------------------------------+-------+
|name |string |null |
| | | |
|# Detailed Table Information| | |
|Database |default | |
|Table |ext2 | |
|Owner |Administrator | |
|Created Time |Wed May 01 21:52:57 IST 2019 | |
|Last Access |Thu Jan 01 05:30:00 IST 1970 | |
|Created By |Spark 2.4.1 | |
|Type |EXTERNAL | |
|Provider |hive | |
|Table Properties |[transient_lastDdlTime=1556727777] | |
|Location |file:/D:/spark-2.4.1-bin-hadoop2.7/bin/spark-warehouse/abc_orc13| |
|Serde Library |org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe | |
|InputFormat |org.apache.hadoop.mapred.TextInputFormat | |
|OutputFormat |org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | |
|Storage Properties |[serialization.format=1] | |
|Partition Provider |Catalog | |
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
