Hyukjin Kwon created SPARK-61168:
------------------------------------
Summary: Log the gRPC error of the call as the exception of the
query context loggers, in the Python client as PySpark does
Key: SPARK-61168
URL: https://issues.apache.org/jira/browse/SPARK-61168
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark Connect builds the exception of an error of the server as it handles
the gRPC error of the call (a grpc _MultiThreadedRendezvous), and raises it
from None. So when the error has a DataFrame or SQL query context, the
DataFrameQueryContextLogger and SQLQueryContextLogger of pyspark.logger log
that gRPC error as the exception of their record: its class, its str (of the
status and the details, the message of the server) and its traceback.
The Python client builds the exception in Rust, where no Python exception is
being handled, so the loggers log an exception of class UnknownException, of
msg "None" and no stacktrace. Apache Spark's
test_dataframe_query_context_logger of test_parity_logger.py fails on this
client (err_msg not in log_json["exception"]["msg"]).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
