Hyukjin Kwon created SPARK-61400:
------------------------------------
Summary: Clone and drop the request of a deep plan on a grown
stack, so that a deep plan or expression does not crash a thread with a small
stack, in Spark Connect Rust
Key: SPARK-61400
URL: https://issues.apache.org/jira/browse/SPARK-61400
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
A follow-up of SPARK-61260, SPARK-61328 and SPARK-61332, found in a post-merge
review. The stack guard of those changes covers the serialization,
Expression::to_proto, render and drop, but not the derived Clone of the request
protos: analyze_plan clones the request on every attempt of with_retry and
execute_plan_reattachable clones it too, on the calling thread. A 256-level
withColumn chain analyzed from a thread with a 512 KiB stack, or a 746-level
one from a 1 MiB thread, crashes the process with SIGBUS in Relation::clone
instead of being sent or raising RecursionError. The merge condition and
actions of MergeIntoWriter are also serialized outside the expression depth
bound, and the comments that give PySpark's bound as 744 or 746 levels do not
hold for either protobuf backend of PySpark.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]