[ 
https://issues.apache.org/jira/browse/SPARK-61416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61416:
-----------------------------------
    Labels: pull-request-available  (was: )

> Raise RecursionError for the expressions of merges, writeTo, function 
> registrations, AutoCDC flows and pickled Columns too deep to send, as PySpark 
> does
> --------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61416
>                 URL: https://issues.apache.org/jira/browse/SPARK-61416
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> SPARK-61328 and SPARK-61332 bound the depth of the expressions and plans that 
> a relation serializes (MAX_EXPRESSION_DEPTH, 745 levels, a RecursionError 
> past it). Expressions that requests serialize outside the plan of a relation 
> have no bound: the condition and the clause conditions and assignments of a 
> merge, the partitioning and the overwrite condition of writeTo, the arguments 
> of a function registration, the expressions of an AutoCDC flow, and a pickled 
> Column (Column.__reduce__, to_plan). A merge or an overwrite of 20,000 levels 
> kills the process with SIGSEGV (a 1,000-level one is sent), and pickling a 
> Column of 20,000 levels takes 21 s and crashes at 150,000. PySpark raises 
> RecursionError for them before any request.
> Fix: serialize those expressions with the same bound 
> (Expression::try_to_proto, Column::try_to_proto), so that they are a 
> RecursionError past 745 levels and are not built past the bound; the 
> drop-in's merge clause methods raise it as PySpark's _build_merge_action does.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to