This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 0d201950 [SPARK-61168][CONNECT][PYTHON] Log the gRPC error of the call 
as the exception of the query context loggers, in the Python client as PySpark 
does
0d201950 is described below

commit 0d201950a207848caa32321d05e82750182e4968
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 16:47:30 2026 +0900

    [SPARK-61168][CONNECT][PYTHON] Log the gRPC error of the call as the 
exception of the query context loggers, in the Python client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61168](https://issues.apache.org/jira/browse/SPARK-61168): PySpark 
Connect builds the exception of an error of the server as it handles the gRPC 
error of the call, a `grpc` `_MultiThreadedRendezvous`, and raises it from 
`None`. So when the error has a DataFrame or SQL query context, the 
`DataFrameQueryContextLogger` and `SQLQueryContextLogger` of `pyspark.logger` 
log that gRPC error as the exception of their record:
    - its class
    - its `str`, of the status and the details (the message of the server)
    - its traceback
    
    The client builds the exception in Rust, where no Python exception is 
handled. Its loggers now log as PySpark Connect's do: an exception of a query 
context logs as it handles a stand-in of that gRPC error, a 
`_MultiThreadedRendezvous` of `pyspark.errors.exceptions.connect`. The stand-in 
has:
    - the status code and message of the error
    - the `str` of `grpc`'s, `<_MultiThreadedRendezvous of RPC that terminated 
with: status = ..., details = "...", debug_error_string = "...">`
    
    The exception is raised from it (`__context__`, `__suppress_context__`), as 
PySpark Connect's. The `debug_error_string` has no address of the peer, which 
the client does not have in Python, and the traceback is of where the exception 
is built.
    
    ### Why are the changes needed?
    
    The loggers logged an exception of class `UnknownException`, of msg `None`, 
with no stacktrace. Apache Spark's `test_dataframe_query_context_logger` of 
`test_parity_logger.py` failed on this client (`err_msg in 
log_json["exception"]["msg"]`).
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. The records of the query context loggers have the exception of PySpark 
4.3.0's records.
    
    ### How was this patch tested?
    
    - A new offline test builds the exception of an error of the server, of a 
DataFrame query context, and checks the record of the 
`DataFrameQueryContextLogger`: its context, and the class, `str` and stacktrace 
of its exception. It also checks the `__context__` of the exception. It fails 
without this change.
    - The record of a division by zero of a DataFrame against a local server is 
that of official PySpark 4.3.0, but for the address of the peer and the frames 
of the stacktrace.
    - Apache Spark's `test_parity_logger.py` passes on this client against a 
local server, all 11 tests. So do `test_parity_traceback.py`, 
`test_parity_errors.py` and `test_parity_dataframe_query_context.py`.
    - The lib tests of both crates (597 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2223 passed, 8 skipped), and the 
live drop-in and Arrow suites (206 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #358 from HyukjinKwon/SPARK-61168-logger-rendezvous.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 python/pyspark/errors/exceptions/connect.py | 42 ++++++++++++++++++++++++-
 python/tests/test_dropin_offline.py         | 49 +++++++++++++++++++++++++++++
 2 files changed, 90 insertions(+), 1 deletion(-)

diff --git a/python/pyspark/errors/exceptions/connect.py 
b/python/pyspark/errors/exceptions/connect.py
index 93e8d9a3..49ac9d22 100644
--- a/python/pyspark/errors/exceptions/connect.py
+++ b/python/pyspark/errors/exceptions/connect.py
@@ -24,6 +24,7 @@ upstream ``_convert_exception`` does. ``grpc`` is optional: 
without it, ``Status
 enum with the members and values of ``grpc.StatusCode``.
 """
 import enum
+import json
 from types import SimpleNamespace
 from typing import Any, Dict, List, Optional
 
@@ -81,6 +82,34 @@ from pyspark.errors.exceptions.base import (
 from pyspark.errors.exceptions.base import TempTableAlreadyExistsException  # 
noqa: F401
 
 
+class _MultiThreadedRendezvous(Exception):
+    """The gRPC error of a call that the server ended with an error, as the 
``grpc`` package's
+    ``_MultiThreadedRendezvous``, of its status code and details: PySpark 
Connect builds its
+    exception of an error of the server as it handles it, so the DataFrame and 
SQL query context
+    loggers log it as the exception, of its class, its ``str`` and its 
traceback."""
+
+    def __init__(self, code: Any, details: str) -> None:
+        super().__init__(code, details)
+        self._code = code
+        self._details = details
+
+    def __str__(self) -> str:
+        grpc_status = self._code.value[0] if isinstance(self._code.value, 
tuple) else self._code
+        debug = "UNKNOWN:Error received from peer {grpc_message:%s, 
grpc_status:%s}" % (
+            json.dumps(self._details),
+            grpc_status,
+        )
+        return (
+            "<_MultiThreadedRendezvous of RPC that terminated with:\n"
+            "\tstatus = %s\n"
+            '\tdetails = "%s"\n'
+            '\tdebug_error_string = "%s"\n'
+            ">" % (self._code, self._details, debug)
+        )
+
+    __repr__ = __str__
+
+
 class SparkConnectException(PySparkException):
     """
     Exception thrown from Spark Connect.
@@ -141,7 +170,18 @@ class SparkConnectGrpcException(SparkConnectException):
         self._contexts: List[BaseQueryContext] = contexts
         self._grpc_status_code = grpc_status_code
         self._breaking_change_info: Optional[Dict[str, Any]] = 
breaking_change_info
-        self._log_exception()
+        if self._contexts:
+            # PySpark Connect builds this exception as it handles the gRPC 
error of the call,
+            # which its query context loggers log as the exception. The Rust 
transport has no
+            # such error in Python, so its stand-in is handled as this logs.
+            try:
+                raise _MultiThreadedRendezvous(self._grpc_status_code, 
self._message)
+            except _MultiThreadedRendezvous as rpc_error:
+                self._log_exception()
+                rpc_error.__traceback__ = None
+                # As PySpark Connect raises this exception from None, as it 
handles that error.
+                self.__context__ = rpc_error
+                self.__suppress_context__ = True
 
     def getSqlState(self) -> Optional[str]:
         if self._sql_state is not None:
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index e4df13f8..c955ada6 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -1226,6 +1226,55 @@ def test_functions_exports_fresh_process():
     assert result.returncode == 0, result.stdout + result.stderr
 
 
+def test_query_context_loggers_log_the_grpc_error_of_the_call_as_pyspark():
+    # As PySpark Connect, which builds the exception of an error of the 
server, of a DataFrame
+    # query context, as it handles the gRPC error of the call: the 
DataFrameQueryContextLogger
+    # logs that error as the exception, and the exception is raised from it 
(from None).
+    import io
+    import logging
+
+    from pyspark.errors import ArithmeticException
+    from pyspark.errors.exceptions.connect import _convert_server_error
+    from pyspark.logger import PySparkLogger
+    from pyspark.logger.logger import JSONFormatter
+
+    logger = PySparkLogger.getLogger("DataFrameQueryContextLogger")
+    stream = io.StringIO()
+    handler = logging.StreamHandler(stream)
+    handler.setFormatter(JSONFormatter())
+    logger.addHandler(handler)
+    context = {
+        "context_type": "DataFrame", "object_type": "", "object_name": "", 
"start_index": -1,
+        "stop_index": -1, "fragment": "__truediv__", "call_site": "app.py:3", 
"summary": "",
+    }
+    try:
+        error = _convert_server_error(
+            classes=["org.apache.spark.SparkArithmeticException", 
"java.lang.ArithmeticException"],
+            sql_state="22012",
+            error_class="DIVIDE_BY_ZERO", reason=None,
+            message="[DIVIDE_BY_ZERO] Division by zero. SQLSTATE: 22012",
+            message_parameters={"config": '"spark.sql.ansi.enabled"'}, 
stacktrace=None,
+            display_server_stacktrace=False, contexts=[context], 
grpc_status_code=13,
+            breaking_change_info=None,
+        )
+    finally:
+        logger.removeHandler(handler)
+    assert isinstance(error, ArithmeticException)
+    record = json.loads(stream.getvalue().strip().splitlines()[-1])
+    assert record["context"] == {
+        "file": "app.py", "line": "3", "fragment": "__truediv__", 
"errorClass": "DIVIDE_BY_ZERO",
+    }
+    exception = record["exception"]
+    assert exception["class"] == "_MultiThreadedRendezvous"
+    assert exception["msg"].startswith(
+        "<_MultiThreadedRendezvous of RPC that terminated with:\n\tstatus = 
StatusCode.INTERNAL\n"
+        '\tdetails = "[DIVIDE_BY_ZERO] Division by zero. SQLSTATE: 22012"\n'
+    )
+    assert set(exception["stacktrace"][0]) == {"class", "method", "file", 
"line"}
+    assert type(error.__context__).__name__ == "_MultiThreadedRendezvous"
+    assert error.__suppress_context__
+
+
 def test_sql_utils_reexports_the_exceptions_of_pyspark(mock_connect):
     # As PySpark's pyspark.sql.utils: the exceptions of pyspark.errors, of 
their import path
     # before it, which the errors of the server are instances of.


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to