This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 9bdad310391 [SPARK-45538][PYTHON][CONNECT] pyspark connect
overwrite_partitions bug
9bdad310391 is described below
commit 9bdad31039134b492caeeba430120d5978a085ee
Author: xieshuaihu <[email protected]>
AuthorDate: Mon Oct 16 17:01:18 2023 +0900
[SPARK-45538][PYTHON][CONNECT] pyspark connect overwrite_partitions bug
### What changes were proposed in this pull request?
Fix a bug in pyspark connect.
DataFrameWriterV2.overwritePartitions set mode as overwrite_partitions
[pyspark/sql/connect/readwriter.py, line 825], but WirteOperationV2 take it as
overwrite_partition [pyspark/sql/connect/plan.py, line 1660]
### Why are the changes needed?
make dataframe.writeTo(table).overwritePartitions() work
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
No test. This bug is very obvious.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #43367 from xieshuaihu/python_connect_overwrite.
Authored-by: xieshuaihu <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
python/pyspark/sql/connect/plan.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python/pyspark/sql/connect/plan.py
b/python/pyspark/sql/connect/plan.py
index 10565b9965a..0121d4c3d57 100644
--- a/python/pyspark/sql/connect/plan.py
+++ b/python/pyspark/sql/connect/plan.py
@@ -1743,7 +1743,7 @@ class WriteOperationV2(LogicalPlan):
plan.write_operation_v2.mode =
proto.WriteOperationV2.Mode.MODE_CREATE
elif wm == "overwrite":
plan.write_operation_v2.mode =
proto.WriteOperationV2.Mode.MODE_OVERWRITE
- elif wm == "overwrite_partition":
+ elif wm == "overwrite_partitions":
plan.write_operation_v2.mode =
proto.WriteOperationV2.Mode.MODE_OVERWRITE_PARTITIONS
elif wm == "append":
plan.write_operation_v2.mode =
proto.WriteOperationV2.Mode.MODE_APPEND
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
