Hyukjin Kwon created SPARK-61416:
------------------------------------

             Summary: Raise RecursionError for the expressions of merges, 
writeTo, function registrations, AutoCDC flows and pickled Columns too deep to 
send, as PySpark does
                 Key: SPARK-61416
                 URL: https://issues.apache.org/jira/browse/SPARK-61416
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


SPARK-61328 and SPARK-61332 bound the depth of the expressions and plans that a 
relation serializes (MAX_EXPRESSION_DEPTH, 745 levels, a RecursionError past 
it). Expressions that requests serialize outside the plan of a relation have no 
bound: the condition and the clause conditions and assignments of a merge, the 
partitioning and the overwrite condition of writeTo, the arguments of a 
function registration, the expressions of an AutoCDC flow, and a pickled Column 
(Column.__reduce__, to_plan). A merge or an overwrite of 20,000 levels kills 
the process with SIGSEGV (a 1,000-level one is sent), and pickling a Column of 
20,000 levels takes 21 s and crashes at 150,000. PySpark raises RecursionError 
for them before any request.

Fix: serialize those expressions with the same bound (Expression::try_to_proto, 
Column::try_to_proto), so that they are a RecursionError past 745 levels and 
are not built past the bound; the drop-in's merge clause methods raise it as 
PySpark's _build_merge_action does.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to