Re: [PR] [SPARK-47828][CONNECT][PYTHON] `DataFrameWriterV2.overwrite` fails with invalid plan [spark]
zhengruifeng commented on PR #46023: URL: https://github.com/apache/spark/pull/46023#issuecomment-2055472673 @dongjoon-hyun this bug was introduced in https://issues.apache.org/jira/browse/SPARK-42002 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47828][CONNECT][PYTHON] `DataFrameWriterV2.overwrite` fails with invalid plan [spark]
zhengruifeng commented on PR #46023: URL: https://github.com/apache/spark/pull/46023#issuecomment-2054964242 merged to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47828][CONNECT][PYTHON] `DataFrameWriterV2.overwrite` fails with invalid plan [spark]
zhengruifeng closed pull request #46023: [SPARK-47828][CONNECT][PYTHON] `DataFrameWriterV2.overwrite` fails with invalid plan URL: https://github.com/apache/spark/pull/46023 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47828][CONNECT][PYTHON] `DataFrameWriterV2.overwrite` fails with invalid plan [spark]
zhengruifeng commented on PR #46023: URL: https://github.com/apache/spark/pull/46023#issuecomment-2051011882 it will need separate PRs for 3.4/3.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47828][CONNECT][PYTHON] `DataFrameWriterV2.overwrite` fails with invalid plan [spark]
zhengruifeng commented on code in PR #46023:
URL: https://github.com/apache/spark/pull/46023#discussion_r1562034412
##
python/pyspark/sql/tests/test_readwriter.py:
##
@@ -252,6 +252,11 @@ def test_create_without_provider(self):
):
df.writeTo("test_table").create()
+def test_table_overwrite(self):
Review Comment:
before this PR, this parity test fails with
```
ERROR [0.846s]: test_table_overwrite
(pyspark.sql.tests.connect.test_parity_readwriter.ReadwriterV2ParityTests.test_table_overwrite)
--
Traceback (most recent call last):
File
"/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/tests/test_readwriter.py",
line 258, in test_table_overwrite
df.writeTo("test_table").overwrite(lit(True))
File
"/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/readwriter.py", line
920, in overwrite
self._spark.client.execute_command(
File
"/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/client/core.py",
line 1054, in execute_command
data, _, _, _, properties = self._execute_and_fetch(req, observations or
{})
File
"/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/client/core.py",
line 1416, in _execute_and_fetch
for response in self._execute_and_fetch_as_iterator(
File
"/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/client/core.py",
line 1393, in _execute_and_fetch_as_iterator
self._handle_error(error)
File
"/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/client/core.py",
line 1663, in _handle_error
self._handle_rpc_error(error)
File
"/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/client/core.py",
line 1735, in _handle_rpc_error
raise convert_exception(
pyspark.errors.exceptions.connect.SparkConnectGrpcException:
(org.apache.spark.sql.connect.common.InvalidPlanInput) Expression with ID: 0 is
not supported
JVM stacktrace:
org.apache.spark.sql.connect.common.InvalidPlanInput
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
