This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push:
new 66b7cb94c373 [SPARK-48089][SS][CONNECT][FOLLOWUP][3.5] Disable Server
Listener failed 3.5 <> 4.0 test
66b7cb94c373 is described below
commit 66b7cb94c373610d9303a85912d7aeee156e4532
Author: Wei Liu <[email protected]>
AuthorDate: Thu Jul 25 09:38:25 2024 +0900
[SPARK-48089][SS][CONNECT][FOLLOWUP][3.5] Disable Server Listener failed
3.5 <> 4.0 test
### What changes were proposed in this pull request?
Disable the listener test. This test would fail after
https://github.com/apache/spark/pull/46921, which is now reverted. The reason
was because with #46921, the server starts a server side python process which
serializes the `StreamingQueryProgress` object with the new
`StreamingQueryProgress` change. But in the client, the client tries to
deserialize `StreamingQueryProgress` use the old `StreamingQueryProgress`
without the change, which caused serde error.
However, as the change is going to spark 4.0, and is considered a generally
good improvement and does more good than harm, we would like to disable this
test to bring back #46921.
### Why are the changes needed?
Unblock bringing back #46921
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
No need
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #47468 from WweiL/3.5-disable-server-listener-test-cross-version.
Authored-by: Wei Liu <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
python/pyspark/sql/tests/connect/streaming/test_parity_listener.py | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/python/pyspark/sql/tests/connect/streaming/test_parity_listener.py
b/python/pyspark/sql/tests/connect/streaming/test_parity_listener.py
index eae2b01c5544..99da04315f0b 100644
--- a/python/pyspark/sql/tests/connect/streaming/test_parity_listener.py
+++ b/python/pyspark/sql/tests/connect/streaming/test_parity_listener.py
@@ -15,6 +15,8 @@
# limitations under the License.
#
import time
+import os
+import unittest
import pyspark.cloudpickle
from pyspark.sql.tests.streaming.test_streaming_listener import
StreamingListenerTestsMixin
@@ -44,6 +46,9 @@ class TestListenerSpark(StreamingQueryListener):
class StreamingListenerParityTests(StreamingListenerTestsMixin,
ReusedConnectTestCase):
+ @unittest.skipIf(
+ "SPARK_SKIP_CONNECT_COMPAT_TESTS" in os.environ, "Failed with
different Client <> Server"
+ )
def test_listener_events(self):
test_listener = TestListenerSpark()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
