This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new 0db78424201 [SPARK-40117][PYTHON][SQL] Convert condition to java in
DataFrameWriterV2.overwrite
0db78424201 is described below
commit 0db78424201cd7b2e2bcffb9de3c2a12a0c67b44
Author: Wenli Looi <wl...@ucalgary.ca>
AuthorDate: Wed Aug 17 15:28:55 2022 +0900
[SPARK-40117][PYTHON][SQL] Convert condition to java in
DataFrameWriterV2.overwrite
### What changes were proposed in this pull request?
Fix DataFrameWriterV2.overwrite() fails to convert the condition parameter
to java. This prevents the function from being called.
It is caused by the following commit that deleted the `_to_java_column`
call instead of fixing it:
https://github.com/apache/spark/commit/a1e459ed9f6777fb8d5a2d09fda666402f9230b9
### Why are the changes needed?
DataFrameWriterV2.overwrite() cannot be called.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manually checked whether the arguments are sent to JVM or not.
Closes #37547 from looi/fix-overwrite.
Authored-by: Wenli Looi <wl...@ucalgary.ca>
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
(cherry picked from commit 46379863ab0dd2ee8fcf1e31e76476ff18397f60)
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
---
python/pyspark/sql/readwriter.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/python/pyspark/sql/readwriter.py b/python/pyspark/sql/readwriter.py
index 760e54831c2..c4c813e56b1 100644
--- a/python/pyspark/sql/readwriter.py
+++ b/python/pyspark/sql/readwriter.py
@@ -1465,6 +1465,7 @@ class DataFrameWriterV2:
Overwrite rows matching the given filter condition with the contents
of the data frame in
the output table.
"""
+ condition = _to_java_column(condition)
self._jwriter.overwrite(condition)
@since(3.1)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
