[
https://issues.apache.org/jira/browse/SPARK-61153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61153:
------------------------------------
Assignee: Hyukjin Kwon
> Run the queries of progress handlers and hooks, and keep an execution
> reattachable when a hook adds reattach options, as PySpark does
> -------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61153
> URL: https://issues.apache.org/jira/browse/SPARK-61153
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> A query run inside a progress handler (registerProgressHandler) or inside
> SparkSession.Hook.on_execute_plan panics with "Cannot start a runtime from
> within a runtime", because the shared runtime's block_on is called from a
> thread already in the runtime's context. Official PySpark runs the nested
> query.
> A hook that appends a request option with reattachable = False makes every
> execution fail with INVALID_HANDLE.OPERATION_NOT_FOUND after its rows are
> read. The reattachable iterator appends its own reattachable = True only when
> no option exists, while PySpark always appends it after the hooks ran.
> Repro: a progress handler running s.range(4).count(), a hook running another
> query, and a hook appending reattach_options.reattachable = False on
> s.range(2).count().
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
