[ 
https://issues.apache.org/jira/browse/SPARK-61332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-61332:
------------------------------------

    Assignee: Hyukjin Kwon

> Send plans of up to 746 levels as PySpark does, and count the plans of 
> subqueries for the depth bound without overflowing the stack
> -----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61332
>                 URL: https://issues.apache.org/jira/browse/SPARK-61332
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> SPARK-61260 bounded the depth of a plan at 256 levels, on the premise that 
> PySpark raises RecursionError from about 245 levels. It does not: official 
> PySpark 4.3.0 serializes plans up to about 746 levels (bisected with 
> _plan.to_proto: 745 filters, 746 withColumns or selects, 747 limits) and 
> raises RecursionError from there, and the server runs plans up to about 510 
> levels. So the Rust client and the Python drop-in reject plans of 257 to 
> about 510 levels that PySpark, the previous master and the server all run, 
> such as Apache's own test_recursion_handling_for_plan_logging (400 
> withColumns, then the schema).
> Also, the thread that SPARK-61260 serializes a deep plan on is sized for the 
> levels of the outer plan only, but the plans of its subqueries (EXISTS, 
> scalar, IN, table arguments) are serialized on it too: an outer plan of about 
> 34 levels with an EXISTS over a plan of 256 levels (Rust) or 300 (drop-in) 
> overflows that thread and aborts the process, as do much shallower ones in 
> debug builds. The previous master ran these plans.
> Raise the bound to 746 levels, as PySpark's, serialize plans on a stack that 
> grows on demand (as SPARK-61328 does for expressions) instead of a sized 
> thread, and count the levels of the plans of subqueries under the plan that 
> has them, as they are sent, for the bound.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to