[
https://issues.apache.org/jira/browse/SPARK-61400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61400:
---------------------------------
Description: A follow-up of SPARK-61260, SPARK-61328 and SPARK-61332, found
in a post-merge review. The stack guard of those changes covers the
serialization, Expression::to_proto, render and drop, but not the derived Clone
of the request protos: analyze_plan clones the request on every attempt of
with_retry and execute_plan_reattachable clones it too, on the calling thread.
A 256-level withColumn chain analyzed from a thread with a 512 KiB stack, or a
746-level one from a 1 MiB thread, crashes the process with SIGBUS in
Relation::clone instead of being sent or raising RecursionError. The comments
that give PySpark's bound as 744 or 746 levels also do not hold for either
protobuf backend of PySpark. (was: A follow-up of SPARK-61260, SPARK-61328 and
SPARK-61332, found in a post-merge review. The stack guard of those changes
covers the serialization, Expression::to_proto, render and drop, but not the
derived Clone of the request protos: analyze_plan clones the request on every
attempt of with_retry and execute_plan_reattachable clones it too, on the
calling thread. A 256-level withColumn chain analyzed from a thread with a 512
KiB stack, or a 746-level one from a 1 MiB thread, crashes the process with
SIGBUS in Relation::clone instead of being sent or raising RecursionError. The
merge condition and actions of MergeIntoWriter are also serialized outside the
expression depth bound, and the comments that give PySpark's bound as 744 or
746 levels do not hold for either protobuf backend of PySpark.)
> Clone and drop the request of a deep plan on a grown stack, so that a deep
> plan or expression does not crash a thread with a small stack, in Spark
> Connect Rust
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61400
> URL: https://issues.apache.org/jira/browse/SPARK-61400
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> A follow-up of SPARK-61260, SPARK-61328 and SPARK-61332, found in a
> post-merge review. The stack guard of those changes covers the serialization,
> Expression::to_proto, render and drop, but not the derived Clone of the
> request protos: analyze_plan clones the request on every attempt of
> with_retry and execute_plan_reattachable clones it too, on the calling
> thread. A 256-level withColumn chain analyzed from a thread with a 512 KiB
> stack, or a 746-level one from a 1 MiB thread, crashes the process with
> SIGBUS in Relation::clone instead of being sent or raising RecursionError.
> The comments that give PySpark's bound as 744 or 746 levels also do not hold
> for either protobuf backend of PySpark.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]