[ 
https://issues.apache.org/jira/browse/SPARK-61400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon updated SPARK-61400:
---------------------------------
    Description: A follow-up of SPARK-61260, SPARK-61328 and SPARK-61332, found 
in a post-merge review. The stack guard of those changes covers the 
serialization, Expression::to_proto, render and drop, but not the derived Clone 
of the request protos: analyze_plan clones the request on every attempt of 
with_retry and execute_plan_reattachable clones it too, on the calling thread. 
A 256-level withColumn chain analyzed from a thread with a 512 KiB stack, or a 
746-level one from a 1 MiB thread, crashes the process with SIGBUS in 
Relation::clone instead of being sent or raising RecursionError. The comments 
that give PySpark's bound as 744 or 746 levels also do not hold for either 
protobuf backend of PySpark.  (was: A follow-up of SPARK-61260, SPARK-61328 and 
SPARK-61332, found in a post-merge review. The stack guard of those changes 
covers the serialization, Expression::to_proto, render and drop, but not the 
derived Clone of the request protos: analyze_plan clones the request on every 
attempt of with_retry and execute_plan_reattachable clones it too, on the 
calling thread. A 256-level withColumn chain analyzed from a thread with a 512 
KiB stack, or a 746-level one from a 1 MiB thread, crashes the process with 
SIGBUS in Relation::clone instead of being sent or raising RecursionError. The 
merge condition and actions of MergeIntoWriter are also serialized outside the 
expression depth bound, and the comments that give PySpark's bound as 744 or 
746 levels do not hold for either protobuf backend of PySpark.)

> Clone and drop the request of a deep plan on a grown stack, so that a deep 
> plan or expression does not crash a thread with a small stack, in Spark 
> Connect Rust
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61400
>                 URL: https://issues.apache.org/jira/browse/SPARK-61400
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>             Fix For: connect-rust-4.3.0
>
>
> A follow-up of SPARK-61260, SPARK-61328 and SPARK-61332, found in a 
> post-merge review. The stack guard of those changes covers the serialization, 
> Expression::to_proto, render and drop, but not the derived Clone of the 
> request protos: analyze_plan clones the request on every attempt of 
> with_retry and execute_plan_reattachable clones it too, on the calling 
> thread. A 256-level withColumn chain analyzed from a thread with a 512 KiB 
> stack, or a 746-level one from a 1 MiB thread, crashes the process with 
> SIGBUS in Relation::clone instead of being sent or raising RecursionError. 
> The comments that give PySpark's bound as 744 or 746 levels also do not hold 
> for either protobuf backend of PySpark.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to