[
https://issues.apache.org/jira/browse/SPARK-9761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15009947#comment-15009947
]
Xin Wu commented on SPARK-9761:
-------------------------------
I can recreate this with the following code
{code}
val conf = new SparkConf().setAppName("TPCDS").setMaster("local")
val sc = new SparkContext(conf)
val hiveContext = new HiveContext(sc)
hiveContext.sql("drop table Orders")
val df = hiveContext.read.json("/home/xwu0226/spark-tables/Orders.json")
df.show()
df.write.saveAsTable("Orders")
hiveContext.sql("ALTER TABLE Orders add columns (z string)")
hiveContext.refreshTable("Orders")
hiveContext.sql("describe extended Orders").show
{code}
Output:
{code}
+----------+-------+
|CustomerID|OrderID|
+----------+-------+
| 452| 1|
+----------+-------+
16:46:28.483 WARN org.apache.hadoop.hive.metastore.HiveMetaStore: Location:
file:/user/hive/warehouse/orders specified for non-external table:orders
+----------+---------+-------+
| col_name|data_type|comment|
+----------+---------+-------+
|CustomerID| bigint| |
| OrderID| bigint| |
+----------+---------+-------+
{code}
I am taking a look.
> Inconsistent metadata handling with ALTER TABLE
> -----------------------------------------------
>
> Key: SPARK-9761
> URL: https://issues.apache.org/jira/browse/SPARK-9761
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.4.1
> Environment: Ubuntu on AWS
> Reporter: Simeon Simeonov
> Labels: hive, sql
>
> Schema changes made with {{ALTER TABLE}} are not shown in {{DESCRIBE TABLE}}.
> The table in question was created with {{HiveContext.read.json()}}.
> Steps:
> # {{alter table dimension_components add columns (z string);}} succeeds.
> # {{describe dimension_components;}} does not show the new column, even after
> restarting spark-sql.
> # A second {{alter table dimension_components add columns (z string);}} fails
> with RROR exec.DDLTask: org.apache.hadoop.hive.ql.metadata.HiveException:
> Duplicate column name: z
> Full spark-sql output
> [here|https://gist.github.com/ssimeonov/d9af4b8bb76b9d7befde].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
