Re: [PR] [SPARK-47828][CONNECT][PYTHON] `DataFrameWriterV2.overwrite` fails with invalid plan [spark]
zhengruifeng commented on PR #46023: URL: https://github.com/apache/spark/pull/46023#issuecomment-2055472673 @dongjoon-hyun this bug was introduced in https://issues.apache.org/jira/browse/SPARK-42002 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47828][CONNECT][PYTHON] `DataFrameWriterV2.overwrite` fails with invalid plan [spark]
zhengruifeng commented on PR #46023: URL: https://github.com/apache/spark/pull/46023#issuecomment-2054964242 merged to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47828][CONNECT][PYTHON] `DataFrameWriterV2.overwrite` fails with invalid plan [spark]
zhengruifeng closed pull request #46023: [SPARK-47828][CONNECT][PYTHON] `DataFrameWriterV2.overwrite` fails with invalid plan URL: https://github.com/apache/spark/pull/46023 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47828][CONNECT][PYTHON] `DataFrameWriterV2.overwrite` fails with invalid plan [spark]
zhengruifeng commented on PR #46023: URL: https://github.com/apache/spark/pull/46023#issuecomment-2051011882 it will need separate PRs for 3.4/3.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47828][CONNECT][PYTHON] `DataFrameWriterV2.overwrite` fails with invalid plan [spark]
zhengruifeng commented on code in PR #46023: URL: https://github.com/apache/spark/pull/46023#discussion_r1562034412 ## python/pyspark/sql/tests/test_readwriter.py: ## @@ -252,6 +252,11 @@ def test_create_without_provider(self): ): df.writeTo("test_table").create() +def test_table_overwrite(self): Review Comment: before this PR, this parity test fails with ``` ERROR [0.846s]: test_table_overwrite (pyspark.sql.tests.connect.test_parity_readwriter.ReadwriterV2ParityTests.test_table_overwrite) -- Traceback (most recent call last): File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/tests/test_readwriter.py", line 258, in test_table_overwrite df.writeTo("test_table").overwrite(lit(True)) File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/readwriter.py", line 920, in overwrite self._spark.client.execute_command( File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/client/core.py", line 1054, in execute_command data, _, _, _, properties = self._execute_and_fetch(req, observations or {}) File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/client/core.py", line 1416, in _execute_and_fetch for response in self._execute_and_fetch_as_iterator( File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/client/core.py", line 1393, in _execute_and_fetch_as_iterator self._handle_error(error) File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/client/core.py", line 1663, in _handle_error self._handle_rpc_error(error) File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/client/core.py", line 1735, in _handle_rpc_error raise convert_exception( pyspark.errors.exceptions.connect.SparkConnectGrpcException: (org.apache.spark.sql.connect.common.InvalidPlanInput) Expression with ID: 0 is not supported JVM stacktrace: org.apache.spark.sql.connect.common.InvalidPlanInput ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org