Lavan Vivekanandasarma created SPARK-61837:
----------------------------------------------
Summary: Avoid per-row Python calls during result materialization
Key: SPARK-61837
URL: https://issues.apache.org/jira/browse/SPARK-61837
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Lavan Vivekanandasarma
The Rust-backed Python client invokes Python Row-construction helpers for each
result row, adding overhead during large result materialization. Replacing
these helpers with native tuple construction removes frequent Python execution
checkpoints and can starve background Python threads and delay Ctrl+C during
conversion. Releasing the GIL during the earlier RPC does not cover this later
conversion phase.
The proposed change uses the native tuple constructor while preserving the
public tuple-subclass Row and shared field-name lists. It adds a cached
Python-frame checkpoint and explicit signal checks every 1,024 Row/nested-value
conversion entries, including values inside large arrays and maps. This
preserves opportunities for thread progress and interruption. The budget bounds
conversion work, not elapsed time; GC, large scalar allocations and
user-defined callbacks can still cause pauses.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]