[
https://issues.apache.org/jira/browse/SPARK-31375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Krishna Chaitanya Chaganti updated SPARK-31375:
---------------------------------------------------
Summary: Overwriting into dynamic partitions is appending data in pyspark
(was: Overwriting into dynamic partitions is appending data)
> Overwriting into dynamic partitions is appending data in pyspark
> ----------------------------------------------------------------
>
> Key: SPARK-31375
> URL: https://issues.apache.org/jira/browse/SPARK-31375
> Project: Spark
> Issue Type: Bug
> Components: PySpark, SQL
> Affects Versions: 2.4.3
> Environment: databricks, s3, EMR, PySpark.
> Reporter: Sai Krishna Chaitanya Chaganti
> Priority: Major
>
> While overwriting data in specific partitions using insertInto , spark is
> appending data to specific partitions though the mode is overwrite. Below
> property is set in config to ensure that we don't overwrite all partitions.
> If the below property is set to static it is truncating and inserting the
> data.
> spark.conf.set('spark.sql.sources.partitionOverwriteMode', 'dynamic')
> df.write.mode('overwrite').format('parquet').insertInto(<db>.<tbl>)
> However if the above statement is changed to
> df.write.mode('overwrite').format('parquet').insertInto(<db>.<tbl>,overwrite=True)
> It starts behaving correct, I mean overwrites the data into specific
> partition.
> It seems though the save mode has been mentioned earlier, precedence is
> given to the parameter set in insertInto method call.
> +_*insertInto(<db>.<tbl>,overwrite=True)*_+
> It is happening in pyspark
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
