This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 0d201950 [SPARK-61168][CONNECT][PYTHON] Log the gRPC error of the call
as the exception of the query context loggers, in the Python client as PySpark
does
0d201950 is described below
commit 0d201950a207848caa32321d05e82750182e4968
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 16:47:30 2026 +0900
[SPARK-61168][CONNECT][PYTHON] Log the gRPC error of the call as the
exception of the query context loggers, in the Python client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61168](https://issues.apache.org/jira/browse/SPARK-61168): PySpark
Connect builds the exception of an error of the server as it handles the gRPC
error of the call, a `grpc` `_MultiThreadedRendezvous`, and raises it from
`None`. So when the error has a DataFrame or SQL query context, the
`DataFrameQueryContextLogger` and `SQLQueryContextLogger` of `pyspark.logger`
log that gRPC error as the exception of their record:
- its class
- its `str`, of the status and the details (the message of the server)
- its traceback
The client builds the exception in Rust, where no Python exception is
handled. Its loggers now log as PySpark Connect's do: an exception of a query
context logs as it handles a stand-in of that gRPC error, a
`_MultiThreadedRendezvous` of `pyspark.errors.exceptions.connect`. The stand-in
has:
- the status code and message of the error
- the `str` of `grpc`'s, `<_MultiThreadedRendezvous of RPC that terminated
with: status = ..., details = "...", debug_error_string = "...">`
The exception is raised from it (`__context__`, `__suppress_context__`), as
PySpark Connect's. The `debug_error_string` has no address of the peer, which
the client does not have in Python, and the traceback is of where the exception
is built.
### Why are the changes needed?
The loggers logged an exception of class `UnknownException`, of msg `None`,
with no stacktrace. Apache Spark's `test_dataframe_query_context_logger` of
`test_parity_logger.py` failed on this client (`err_msg in
log_json["exception"]["msg"]`).
### Does this PR introduce _any_ user-facing change?
Yes. The records of the query context loggers have the exception of PySpark
4.3.0's records.
### How was this patch tested?
- A new offline test builds the exception of an error of the server, of a
DataFrame query context, and checks the record of the
`DataFrameQueryContextLogger`: its context, and the class, `str` and stacktrace
of its exception. It also checks the `__context__` of the exception. It fails
without this change.
- The record of a division by zero of a DataFrame against a local server is
that of official PySpark 4.3.0, but for the address of the peer and the frames
of the stacktrace.
- Apache Spark's `test_parity_logger.py` passes on this client against a
local server, all 11 tests. So do `test_parity_traceback.py`,
`test_parity_errors.py` and `test_parity_dataframe_query_context.py`.
- The lib tests of both crates (597 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2223 passed, 8 skipped), and the
live drop-in and Arrow suites (206 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #358 from HyukjinKwon/SPARK-61168-logger-rendezvous.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
python/pyspark/errors/exceptions/connect.py | 42 ++++++++++++++++++++++++-
python/tests/test_dropin_offline.py | 49 +++++++++++++++++++++++++++++
2 files changed, 90 insertions(+), 1 deletion(-)
diff --git a/python/pyspark/errors/exceptions/connect.py
b/python/pyspark/errors/exceptions/connect.py
index 93e8d9a3..49ac9d22 100644
--- a/python/pyspark/errors/exceptions/connect.py
+++ b/python/pyspark/errors/exceptions/connect.py
@@ -24,6 +24,7 @@ upstream ``_convert_exception`` does. ``grpc`` is optional:
without it, ``Status
enum with the members and values of ``grpc.StatusCode``.
"""
import enum
+import json
from types import SimpleNamespace
from typing import Any, Dict, List, Optional
@@ -81,6 +82,34 @@ from pyspark.errors.exceptions.base import (
from pyspark.errors.exceptions.base import TempTableAlreadyExistsException #
noqa: F401
+class _MultiThreadedRendezvous(Exception):
+ """The gRPC error of a call that the server ended with an error, as the
``grpc`` package's
+ ``_MultiThreadedRendezvous``, of its status code and details: PySpark
Connect builds its
+ exception of an error of the server as it handles it, so the DataFrame and
SQL query context
+ loggers log it as the exception, of its class, its ``str`` and its
traceback."""
+
+ def __init__(self, code: Any, details: str) -> None:
+ super().__init__(code, details)
+ self._code = code
+ self._details = details
+
+ def __str__(self) -> str:
+ grpc_status = self._code.value[0] if isinstance(self._code.value,
tuple) else self._code
+ debug = "UNKNOWN:Error received from peer {grpc_message:%s,
grpc_status:%s}" % (
+ json.dumps(self._details),
+ grpc_status,
+ )
+ return (
+ "<_MultiThreadedRendezvous of RPC that terminated with:\n"
+ "\tstatus = %s\n"
+ '\tdetails = "%s"\n'
+ '\tdebug_error_string = "%s"\n'
+ ">" % (self._code, self._details, debug)
+ )
+
+ __repr__ = __str__
+
+
class SparkConnectException(PySparkException):
"""
Exception thrown from Spark Connect.
@@ -141,7 +170,18 @@ class SparkConnectGrpcException(SparkConnectException):
self._contexts: List[BaseQueryContext] = contexts
self._grpc_status_code = grpc_status_code
self._breaking_change_info: Optional[Dict[str, Any]] =
breaking_change_info
- self._log_exception()
+ if self._contexts:
+ # PySpark Connect builds this exception as it handles the gRPC
error of the call,
+ # which its query context loggers log as the exception. The Rust
transport has no
+ # such error in Python, so its stand-in is handled as this logs.
+ try:
+ raise _MultiThreadedRendezvous(self._grpc_status_code,
self._message)
+ except _MultiThreadedRendezvous as rpc_error:
+ self._log_exception()
+ rpc_error.__traceback__ = None
+ # As PySpark Connect raises this exception from None, as it
handles that error.
+ self.__context__ = rpc_error
+ self.__suppress_context__ = True
def getSqlState(self) -> Optional[str]:
if self._sql_state is not None:
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index e4df13f8..c955ada6 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -1226,6 +1226,55 @@ def test_functions_exports_fresh_process():
assert result.returncode == 0, result.stdout + result.stderr
+def test_query_context_loggers_log_the_grpc_error_of_the_call_as_pyspark():
+ # As PySpark Connect, which builds the exception of an error of the
server, of a DataFrame
+ # query context, as it handles the gRPC error of the call: the
DataFrameQueryContextLogger
+ # logs that error as the exception, and the exception is raised from it
(from None).
+ import io
+ import logging
+
+ from pyspark.errors import ArithmeticException
+ from pyspark.errors.exceptions.connect import _convert_server_error
+ from pyspark.logger import PySparkLogger
+ from pyspark.logger.logger import JSONFormatter
+
+ logger = PySparkLogger.getLogger("DataFrameQueryContextLogger")
+ stream = io.StringIO()
+ handler = logging.StreamHandler(stream)
+ handler.setFormatter(JSONFormatter())
+ logger.addHandler(handler)
+ context = {
+ "context_type": "DataFrame", "object_type": "", "object_name": "",
"start_index": -1,
+ "stop_index": -1, "fragment": "__truediv__", "call_site": "app.py:3",
"summary": "",
+ }
+ try:
+ error = _convert_server_error(
+ classes=["org.apache.spark.SparkArithmeticException",
"java.lang.ArithmeticException"],
+ sql_state="22012",
+ error_class="DIVIDE_BY_ZERO", reason=None,
+ message="[DIVIDE_BY_ZERO] Division by zero. SQLSTATE: 22012",
+ message_parameters={"config": '"spark.sql.ansi.enabled"'},
stacktrace=None,
+ display_server_stacktrace=False, contexts=[context],
grpc_status_code=13,
+ breaking_change_info=None,
+ )
+ finally:
+ logger.removeHandler(handler)
+ assert isinstance(error, ArithmeticException)
+ record = json.loads(stream.getvalue().strip().splitlines()[-1])
+ assert record["context"] == {
+ "file": "app.py", "line": "3", "fragment": "__truediv__",
"errorClass": "DIVIDE_BY_ZERO",
+ }
+ exception = record["exception"]
+ assert exception["class"] == "_MultiThreadedRendezvous"
+ assert exception["msg"].startswith(
+ "<_MultiThreadedRendezvous of RPC that terminated with:\n\tstatus =
StatusCode.INTERNAL\n"
+ '\tdetails = "[DIVIDE_BY_ZERO] Division by zero. SQLSTATE: 22012"\n'
+ )
+ assert set(exception["stacktrace"][0]) == {"class", "method", "file",
"line"}
+ assert type(error.__context__).__name__ == "_MultiThreadedRendezvous"
+ assert error.__suppress_context__
+
+
def test_sql_utils_reexports_the_exceptions_of_pyspark(mock_connect):
# As PySpark's pyspark.sql.utils: the exceptions of pyspark.errors, of
their import path
# before it, which the errors of the server are instances of.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]