Lavan Vivekanandasarma created SPARK-61838:
----------------------------------------------
Summary: Reduce peak memory while collecting rows
Key: SPARK-61838
URL: https://issues.apache.org/jira/browse/SPARK-61838
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Lavan Vivekanandasarma
The Rust-backed Python client retains the entire native result vector while
constructing Python Rows. Large string and binary payloads therefore remain
alive in both native and Python representations during collection, causing
avoidable peak memory use.
The proposed change consumes native rows by value and releases each native
payload after its Python Row is constructed. It applies to collect, first/head,
take and tail, preserving returned values, Row behavior and public method
signatures. Iterator and configuration-read behavior remain unchanged.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]