[
https://issues.apache.org/jira/browse/SPARK-31375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Krishna Chaitanya Chaganti updated SPARK-31375:
---------------------------------------------------
Description:
While overwriting data in specific partitions using insertInto , spark is
appending data to specific partitions though the mode is overwrite. Below
property is set in config to ensure that we don't overwrite all partitions. If
the below property is set to static it is truncating and inserting the data.
_spark.conf.set('spark.sql.sources.partitionOverwriteMode', 'dynamic')_
_df.write.mode('overwrite').format('parquet').insertInto(<db>.<tbl>)_
However if the above statement is changed to
_df.write.mode('overwrite').format('parquet').insertInto(<db>.<tbl>,overwrite=True)_
It starts behaving correct, I mean overwrites the data into specific
partition.
It seems though the save mode has been mentioned earlier, precedence is given
to the parameter set in insertInto method call.
+_*insertInto(<db>.<tbl>,overwrite=True)*_+
It is happening in pyspark
was:
While overwriting data in specific partitions using insertInto , spark is
appending data to specific partitions though the mode is overwrite. Below
property is set in config to ensure that we don't overwrite all partitions. If
the below property is set to static it is truncating and inserting the data.
spark.conf.set('spark.sql.sources.partitionOverwriteMode', 'dynamic')
df.write.mode('overwrite').format('parquet').insertInto(<db>.<tbl>)
However if the above statement is changed to
df.write.mode('overwrite').format('parquet').insertInto(<db>.<tbl>,overwrite=True)
It starts behaving correct, I mean overwrites the data into specific
partition.
It seems though the save mode has been mentioned earlier, precedence is given
to the parameter set in insertInto method call.
+_*insertInto(<db>.<tbl>,overwrite=True)*_+
It is happening in pyspark
> Overwriting into dynamic partitions is appending data in pyspark
> ----------------------------------------------------------------
>
> Key: SPARK-31375
> URL: https://issues.apache.org/jira/browse/SPARK-31375
> Project: Spark
> Issue Type: Bug
> Components: PySpark, SQL
> Affects Versions: 2.4.3
> Environment: databricks, s3, EMR, PySpark.
> Reporter: Sai Krishna Chaitanya Chaganti
> Priority: Major
>
> While overwriting data in specific partitions using insertInto , spark is
> appending data to specific partitions though the mode is overwrite. Below
> property is set in config to ensure that we don't overwrite all partitions.
> If the below property is set to static it is truncating and inserting the
> data.
> _spark.conf.set('spark.sql.sources.partitionOverwriteMode', 'dynamic')_
> _df.write.mode('overwrite').format('parquet').insertInto(<db>.<tbl>)_
> However if the above statement is changed to
> _df.write.mode('overwrite').format('parquet').insertInto(<db>.<tbl>,overwrite=True)_
> It starts behaving correct, I mean overwrites the data into specific
> partition.
> It seems though the save mode has been mentioned earlier, precedence is
> given to the parameter set in insertInto method call.
> +_*insertInto(<db>.<tbl>,overwrite=True)*_+
> It is happening in pyspark
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
