Lavan Vivekanandasarma created SPARK-61838:
----------------------------------------------

             Summary: Reduce peak memory while collecting rows
                 Key: SPARK-61838
                 URL: https://issues.apache.org/jira/browse/SPARK-61838
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Lavan Vivekanandasarma


The Rust-backed Python client retains the entire native result vector while 
constructing Python Rows. Large string and binary payloads therefore remain 
alive in both native and Python representations during collection, causing 
avoidable peak memory use.

The proposed change consumes native rows by value and releases each native 
payload after its Python Row is constructed. It applies to collect, first/head, 
take and tail, preserving returned values, Row behavior and public method 
signatures. Iterator and configuration-read behavior remain unchanged.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to