Lavan Vivekanandasarma created SPARK-61837:
----------------------------------------------

             Summary: Avoid per-row Python calls during result materialization
                 Key: SPARK-61837
                 URL: https://issues.apache.org/jira/browse/SPARK-61837
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Lavan Vivekanandasarma


The Rust-backed Python client invokes Python Row-construction helpers for each 
result row, adding overhead during large result materialization. Replacing 
these helpers with native tuple construction removes frequent Python execution 
checkpoints and can starve background Python threads and delay Ctrl+C during 
conversion. Releasing the GIL during the earlier RPC does not cover this later 
conversion phase.

The proposed change uses the native tuple constructor while preserving the 
public tuple-subclass Row and shared field-name lists. It adds a cached 
Python-frame checkpoint and explicit signal checks every 1,024 Row/nested-value 
conversion entries, including values inside large arrays and maps. This 
preserves opportunities for thread progress and interruption. The budget bounds 
conversion work, not elapsed time; GC, large scalar allocations and 
user-defined callbacks can still cause pauses.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to