[
https://issues.apache.org/jira/browse/SPARK-61188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61188:
------------------------------------
Assignee: Hyukjin Kwon
> Raise RecursionError instead of crashing when progress handlers run queries
> without end, in the Python client as PySpark does
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61188
> URL: https://issues.apache.org/jira/browse/SPARK-61188
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> An unguarded progress handler that runs a query (for example
> `spark.registerProgressHandler(lambda **kw: spark.range(4).count() if not
> kw["done"] else None)`) recurses: the nested query reports progress, which
> calls the handler again. Since SPARK-61153 the nested query runs, so the
> recursion has no end and the native stack of the process overflows; the
> interpreter dies with SIGSEGV (exit 139) and no Python exception. PySpark
> raises RecursionError from the nested query (surfaced as
> SparkConnectGrpcException "Exception serializing request!") and the process
> survives. This change bounds the nesting of progress handlers and hooks that
> run queries, and raises RecursionError past the bound, so the process
> survives too.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
