Hyukjin Kwon created SPARK-61328:
------------------------------------
Summary: Raise RecursionError instead of crashing the process on a
deep expression, in the Python client as PySpark does
Key: SPARK-61328
URL: https://issues.apache.org/jira/browse/SPARK-61328
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
SPARK-61260 bounded the depth of a plan, but not the depth of an expression.
Serializing an expression recursed once per level with no bound, so in the
Python client (and the Rust API) a deep expression crashed the process. Against
an OSS Spark 4.3.0 Connect server, select((col("id") + 1 + ... +
1).alias("x")).collect():
- PySpark 4.3.0 raises RecursionError from 744 additions, before any request,
and the process lives.
- master: 744 to a few thousand additions were sent and rejected by the server
with "[Unknown error] Application error processing RPC", and from about 30,000
the process died with SIGSEGV, on the main thread of Python. The serialized
proto of such an expression is also dropped and encoded by recursing.
Serialize and render expressions on a stack that grows on demand (stacker,
already in Cargo.lock), and raise RecursionError for a request whose plan has
an expression deeper than 745 levels, as PySpark does, without serializing the
expression past the bound.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]