yangz created SPARK-24194:
-----------------------------
Summary: HadoopFsRelation cannot overwrite a path that is also
being read from
Key: SPARK-24194
URL: https://issues.apache.org/jira/browse/SPARK-24194
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 2.4.0
Environment: spark master
Reporter: yangz
Fix For: 2.4.0
When
{code:java}
INSERT OVERWRITE TABLE territory_count_compare select * from
territory_count_compare where shop_count!=real_shop_count
{code}
And territory_count_compare is a table with parquet, there will be a error
Cannot overwrite a path that is also being read from
And in file MetastoreDataSourceSuite.scala, there have a test case
{code:java}
table(tableName).write.mode(SaveMode.Overwrite).insertInto(tableName)
{code}
But when the table territory_count_compare is a common hive table, there is no
error.
So I think the reason is when insert overwrite into hadoopfs relation with
static partition, it first delete the partition in the output. But it should be
the time when the job commited.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
