[
https://issues.apache.org/jira/browse/SPARK-44239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wan Kun updated SPARK-44239:
----------------------------
Attachment: image-2023-06-29-12-58-12-256.png
> Reclaim memory allocated by huge column vector
> ----------------------------------------------
>
> Key: SPARK-44239
> URL: https://issues.apache.org/jira/browse/SPARK-44239
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.5.0
> Reporter: Wan Kun
> Priority: Major
> Attachments: image-2023-06-29-12-58-12-256.png
>
>
> When spark read data files into WritableColumnVectors, the memory allocated
> by the WritableColumnVectors will not be free until the
> VectorizedColumnReader is finished.
> It will save the memory allocation time though reusing the allocated array
> object. But it will also occupy too many unused memory after the current
> large vector batch is already read.
>
> Add a vector reserve policy for this scenario, which will use the allocated
> array object for small column vectors and free up the memory for huge column
> vectors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
