[
https://issues.apache.org/jira/browse/SPARK-61416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61416.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 492
[https://github.com/apache/spark-connect-rust/pull/492]
> Raise RecursionError for the expressions of merges, writeTo, function
> registrations, AutoCDC flows and pickled Columns too deep to send, as PySpark
> does
> --------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61416
> URL: https://issues.apache.org/jira/browse/SPARK-61416
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> SPARK-61328 and SPARK-61332 bound the depth of the expressions and plans that
> a relation serializes (MAX_EXPRESSION_DEPTH, 745 levels, a RecursionError
> past it). Expressions that requests serialize outside the plan of a relation
> have no bound: the condition and the clause conditions and assignments of a
> merge, the partitioning and the overwrite condition of writeTo, the arguments
> of a function registration, the expressions of an AutoCDC flow, and a pickled
> Column (Column.__reduce__, to_plan). A merge or an overwrite of 20,000 levels
> kills the process with SIGSEGV (a 1,000-level one is sent), and pickling a
> Column of 20,000 levels takes 21 s and crashes at 150,000. PySpark raises
> RecursionError for them before any request.
> Fix: serialize those expressions with the same bound
> (Expression::try_to_proto, Column::try_to_proto), so that they are a
> RecursionError past 745 levels and are not built past the bound; the
> drop-in's merge clause methods raise it as PySpark's _build_merge_action does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]