[
https://issues.apache.org/jira/browse/SPARK-61332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61332:
------------------------------------
Assignee: Hyukjin Kwon
> Send plans of up to 746 levels as PySpark does, and count the plans of
> subqueries for the depth bound without overflowing the stack
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61332
> URL: https://issues.apache.org/jira/browse/SPARK-61332
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> SPARK-61260 bounded the depth of a plan at 256 levels, on the premise that
> PySpark raises RecursionError from about 245 levels. It does not: official
> PySpark 4.3.0 serializes plans up to about 746 levels (bisected with
> _plan.to_proto: 745 filters, 746 withColumns or selects, 747 limits) and
> raises RecursionError from there, and the server runs plans up to about 510
> levels. So the Rust client and the Python drop-in reject plans of 257 to
> about 510 levels that PySpark, the previous master and the server all run,
> such as Apache's own test_recursion_handling_for_plan_logging (400
> withColumns, then the schema).
> Also, the thread that SPARK-61260 serializes a deep plan on is sized for the
> levels of the outer plan only, but the plans of its subqueries (EXISTS,
> scalar, IN, table arguments) are serialized on it too: an outer plan of about
> 34 levels with an EXISTS over a plan of 256 levels (Rust) or 300 (drop-in)
> overflows that thread and aborts the process, as do much shallower ones in
> debug builds. The previous master ran these plans.
> Raise the bound to 746 levels, as PySpark's, serialize plans on a stack that
> grows on demand (as SPARK-61328 does for expressions) instead of a sized
> thread, and count the levels of the plans of subqueries under the plan that
> has them, as they are sent, for the bound.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]