[
https://issues.apache.org/jira/browse/SPARK-27617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sujith Chacko updated SPARK-27617:
----------------------------------
Description:
In spark whenever user specifies location uri in create table without external
keyword the table is treated as external table .
Because of this behavior following following problems has been observed
a) user will not able to set an external location for a managed table.
b) compatibility issue with hive/impala where the system allow managed table to
specify location uri if user created table without 'EXTERNAL' keyword.
scala> spark.sql("""CREATE TABLE IF NOT EXISTS ext2 (name STRING) LOCATION
'D:/spark-2.4.1-bin-hadoop2.7/bin/spark-warehouse/abc_orc13'""");
-chgrp: 'HTIPL-23270\None' does not match expected pattern for group
Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...
res15: org.apache.spark.sql.DataFrame = []
scala> spark.sql("desc formatted ext2").show(false)
+-----------------------------+---------------------------------------------------------------++-------
|col_name|data_type|comment|
+-----------------------------+---------------------------------------------------------------++-------
|name|string|null|
| | | |
| # Detailed Table Information| | |
|Database|default| |
|Table|ext2| |
|Owner|Administrator| |
|Created Time|Wed May 01 21:52:57 IST 2019| |
|Last Access|Thu Jan 01 05:30:00 IST 1970| |
|Created By|Spark 2.4.1| |
|Type|EXTERNAL| |
|Provider|hive| |
|Table Properties|[transient_lastDdlTime=1556727777]| |
|Location|file:/D:/spark-2.4.1-bin-hadoop2.7/bin/spark-warehouse/abc_orc13| |
|Serde Library|org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe| |
|InputFormat|org.apache.hadoop.mapred.TextInputFormat| |
|OutputFormat|org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat| |
|Storage Properties|[serialization.format=1]| |
|Partition Provider|Catalog| |
was:
In spark whenever user specifies location uri in create table without external
keyword the table is treated as external table .
Because of this behavior following following problems has been observed
a) user will not able to set an external location for a managed table.
b) compatibility issue with hive impala where the system allow managed table
to specify location uri if user created table without 'EXTERNAL' keyword.
scala> spark.sql("""CREATE TABLE IF NOT EXISTS ext2 (name STRING) LOCATION
'D:/spark-2.4.1-bin-hadoop2.7/bin/spark-warehouse/abc_orc13'""");
-chgrp: 'HTIPL-23270\None' does not match expected pattern for group
Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...
res15: org.apache.spark.sql.DataFrame = []
scala> spark.sql("desc formatted ext2").show(false)
+----------------------------+----------------------------------------------------------------+-------+
|col_name |data_type |comment|
+----------------------------+----------------------------------------------------------------+-------+
|name |string |null |
| | | |
|# Detailed Table Information| | |
|Database |default | |
|Table |ext2 | |
|Owner |Administrator | |
|Created Time |Wed May 01 21:52:57 IST 2019 | |
|Last Access |Thu Jan 01 05:30:00 IST 1970 | |
|Created By |Spark 2.4.1 | |
|Type |EXTERNAL | |
|Provider |hive | |
|Table Properties |[transient_lastDdlTime=1556727777] | |
|Location |file:/D:/spark-2.4.1-bin-hadoop2.7/bin/spark-warehouse/abc_orc13| |
|Serde Library |org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe | |
|InputFormat |org.apache.hadoop.mapred.TextInputFormat | |
|OutputFormat |org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | |
|Storage Properties |[serialization.format=1] | |
|Partition Provider |Catalog | |
> Not able to specify LOCATION for internal table
> -----------------------------------------------
>
> Key: SPARK-27617
> URL: https://issues.apache.org/jira/browse/SPARK-27617
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.2.0, 2.3.0, 2.4.0
> Reporter: Sujith Chacko
> Priority: Major
>
> In spark whenever user specifies location uri in create table without
> external keyword the table is treated as external table .
> Because of this behavior following following problems has been observed
> a) user will not able to set an external location for a managed table.
> b) compatibility issue with hive/impala where the system allow managed table
> to specify location uri if user created table without 'EXTERNAL' keyword.
>
> scala> spark.sql("""CREATE TABLE IF NOT EXISTS ext2 (name STRING) LOCATION
> 'D:/spark-2.4.1-bin-hadoop2.7/bin/spark-warehouse/abc_orc13'""");
> -chgrp: 'HTIPL-23270\None' does not match expected pattern for group
> Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...
> res15: org.apache.spark.sql.DataFrame = []
> scala> spark.sql("desc formatted ext2").show(false)
>
> +-----------------------------+---------------------------------------------------------------++-------
> |col_name|data_type|comment|
> +-----------------------------+---------------------------------------------------------------++-------
> |name|string|null|
> | | | |
> | # Detailed Table Information| | |
> |Database|default| |
> |Table|ext2| |
> |Owner|Administrator| |
> |Created Time|Wed May 01 21:52:57 IST 2019| |
> |Last Access|Thu Jan 01 05:30:00 IST 1970| |
> |Created By|Spark 2.4.1| |
> |Type|EXTERNAL| |
> |Provider|hive| |
> |Table Properties|[transient_lastDdlTime=1556727777]| |
> |Location|file:/D:/spark-2.4.1-bin-hadoop2.7/bin/spark-warehouse/abc_orc13| |
> |Serde Library|org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe| |
> |InputFormat|org.apache.hadoop.mapred.TextInputFormat| |
> |OutputFormat|org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat| |
> |Storage Properties|[serialization.format=1]| |
> |Partition Provider|Catalog| |
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
