Sai Krishna Chaitanya Chaganti created SPARK-31375:
------------------------------------------------------
Summary: Overwriting into dynamic partitions is appending data
Key: SPARK-31375
URL: https://issues.apache.org/jira/browse/SPARK-31375
Project: Spark
Issue Type: Bug
Components: PySpark, SQL
Affects Versions: 2.4.3
Environment: databricks, s3, EMR, PySpark.
Reporter: Sai Krishna Chaitanya Chaganti
While overwriting data in specific partitions using insertInto , spark is
appending data to specific partitions though the mode is overwrite. Below
property is set in config to ensure that we don't overwrite all partitions. If
the below property is set to static it is truncating and inserting the data.
spark.conf.set('spark.sql.sources.partitionOverwriteMode', 'dynamic')
df.write.mode('overwrite').format('parquet').insertInto(<db>.<tbl>)
However if the above statement is changed to
df.write.mode('overwrite').format('parquet').insertInto(<db>.<tbl>,overwrite=True)
It starts behaving correct, I mean overwrites the data into specific
partition.
It seems though the save mode has been mentioned earlier, precedence is given
to the parameter set in insertInto method call.
+_*insertInto(<db>.<tbl>,overwrite=True)*_+
It is happening in pyspark
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
