[
https://issues.apache.org/jira/browse/SPARK-11748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15007828#comment-15007828
]
Xin Wu commented on SPARK-11748:
--------------------------------
I can recreate it for parquet format.. but not for orc or default text format..
Digging in more.
{code}
scala> sqlContext.sql("create table spark_11748 (id int, str string) stored as
orc")
res15: org.apache.spark.sql.DataFrame = [result: string]
scala> df.write.format("orc").mode(SaveMode.Append).saveAsTable("spark_11748")
scala> sqlContext.sql("select * from spark_11748").show
+---+---+
| id|str|
+---+---+
| 1|abd|
+---+---+
scala> sqlContext.sql("alter table spark_11748 change id i_d int")
res18: org.apache.spark.sql.DataFrame = [result: string]
scala> sqlContext.sql("select * from spark_11748").show
+---+---+
|i_d|str|
+---+---+
| 1|abd|
+---+---+
scala> sqlContext.sql("create table spark_11748_1 (id int, str string)")
res21: org.apache.spark.sql.DataFrame = [result: string]
scala> df.write.mode(SaveMode.Append).saveAsTable("spark_11748_1")
scala> sqlContext.sql("select * from spark_11748_1").show
+---+---+
| id|str|
+---+---+
| 1|abd|
+---+---+
scala> sqlContext.sql("alter table spark_11748_1 change id i_d int")
res24: org.apache.spark.sql.DataFrame = [result: string]
scala> sqlContext.sql("select * from spark_11748_1").show
+---+---+
|i_d|str|
+---+---+
| 1|abd|
+---+---+
{code}
> Result is null after alter column name of table stored as Parquet
> ------------------------------------------------------------------
>
> Key: SPARK-11748
> URL: https://issues.apache.org/jira/browse/SPARK-11748
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.5.1
> Reporter: pin_zhang
>
> 1. Test with the following code
> hctx.sql(" create table " + table + " (id int, str string) STORED AS
> PARQUET ")
> val df = hctx.jsonFile("g:/vip.json")
> df.write.format("parquet").mode(SaveMode.Append).saveAsTable(table)
> hctx.sql(" select * from " + table).show()
> // alter table
> val alter = "alter table " + table + " CHANGE id i_d int "
> hctx.sql(alter)
>
> hctx.sql(" select * from " + table).show()
> 2. Result
> after change table column name, data in null for the changed column
> Result before alter table
> +---+---+
> | id|str|
> +---+---+
> | 1| s1|
> | 2| s2|
> +---+---+
> Result after alter table
> +----+---+
> | i_d|str|
> +----+---+
> |null| s1|
> |null| s2|
> +----+---+
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
