Hyukjin Kwon created SPARK-62130:
------------------------------------
Summary: Decode the responses of the server within upb's limit of
nested messages, of PySpark's error, without a crash at any depth or on threads
of small stacks
Key: SPARK-62130
URL: https://issues.apache.org/jira/browse/SPARK-62130
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
The Rust client decodes the responses of the server with prost, whose
no-recursion-limit feature the workspace enables for the client's own deep
plans, so the responses decode at any depth:
- A response nested very deeply, such as the schema of a struct nested 10^4
levels or a literal of 10^5, overflows the stack of the decode and the process
ends with SIGSEGV, in the Python client and the Rust API alike.
- A response of more than 100 levels of nested messages decodes, where PySpark
4.3.0, whose grpc deserializer is upb's FromString, raises the
SparkConnectGrpcException of INTERNAL "Exception deserializing response!": a
struct column nested 32 levels deep, say.
- The conversions of deep responses and types recurse on the calling thread's
stack, so calls on threads of small stacks (threading.stack_size of 256 KiB)
crash, even a trivial spark.range(1).schema.
Expected: as pyspark-client, the responses decode within upb's limit of 100
nested messages (65535 once upb's SetAllowOversizeProtos(True) is called), of
PySpark's error in the drop-in and of prost's error in the Rust API, without a
crash at any depth, and calls on threads of small stacks work as PySpark's.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]