[
https://issues.apache.org/jira/browse/SPARK-15538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Or updated SPARK-15538:
------------------------------
Description:
Truncate table does not seems to work on data source table. It returns success
without any error , but table is not truncated.
Repro:
{code}
val df = Seq((1 , "john", "CA") ,(2,"Mike", "NY"), (3, "Robert",
"CA")).toDF("id", "name", "state")
df.write.format("parquet").partitionBy("state").saveAsTable("emp")
scala> sql("truncate table emp")
res8: org.apache.spark.sql.DataFrame = []
scala> sql("select * from emp").show() // FileNotFoundException
{code}
was:
Truncate table does not seems to work on data source table. It returns success
without any error , but table is not truncated.
Repro:
{code}
val df = Seq((1 , "john", "CA") ,(2,"Mike", "NY"), (3, "Robert",
"CA")).toDF("id", "name", "state")
df.write.format("parquet").partitionBy("state").saveAsTable("emp")
scala> sql("truncate table emp")
res8: org.apache.spark.sql.DataFrame = []
scala> sql("select * from emp").show ;
+---+------+-----+
| id| name|state|
+---+------+-----+
| 3|Robert| CA|
| 1| john| CA|
| 2| Mike| NY|
+---+------+-----+
{code}
The select should have returned no results.
By scanning through the code I found some of the other DDL commands like
LOAD DATA , and SHOW PARTITIONS are not allowed for data source table and they
raise error.
It Might be good to throw error until the truncate table works with data
source table also.
> Truncate table does not work on data source table
> -------------------------------------------------
>
> Key: SPARK-15538
> URL: https://issues.apache.org/jira/browse/SPARK-15538
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Suresh Thalamati
> Assignee: Andrew Or
> Priority: Minor
>
> Truncate table does not seems to work on data source table. It returns
> success without any error , but table is not truncated.
> Repro:
> {code}
> val df = Seq((1 , "john", "CA") ,(2,"Mike", "NY"), (3, "Robert",
> "CA")).toDF("id", "name", "state")
> df.write.format("parquet").partitionBy("state").saveAsTable("emp")
> scala> sql("truncate table emp")
> res8: org.apache.spark.sql.DataFrame = []
> scala> sql("select * from emp").show() // FileNotFoundException
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
