[
https://issues.apache.org/jira/browse/ARROW-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16728819#comment-16728819
]
Tanya Schlusser commented on ARROW-3324:
----------------------------------------
I could not reproduce either of the two GitHub issues above, but could identify
a leak using {{memory_profiler}} on the stackoverflow code (copied
[this|https://github.com/apache/arrow/blob/master/python/scripts/test_leak.py])
I observed that {{FileSerializer.properties_.use_count()}} increments more than
expected whenever {{FileSerializer.AppendRowGroup}} is called. The offending
line is {{FileSerializer.metadata_->AppendRowGroup()}}. I believe that the
count should only go up once per new row group, instead of once per column plus
once per row group.
I think the root cause is that in
{{RowGroupMetaDataBuilder::RowGroupMetaDataBuilderImpl.Finish}}, the vector of
{{column_builders_}} ought to be reset and cleared each time before it is
repopulated. I hope to submit a pull request for this even though it may not
address all of the issues stated here. Since the GitHub issues were about
memory leaks on "read", and the fix is related only to "write", this
observation certainly doesn't address everything in this JIRA issue.
Even after the fix I'll post, my memory_profiler code still shows an increase
in memory use upon additional calls to {{pq.ParquetWriter.write_table}}, which
I think is OK because the row group is incrementing with each write too. So I
may be wrong or have still missed something. Regardless, I hope these notes are
useful to someone.
> [Python] Users reporting memory leaks using pa.pq.ParquetDataset
> ----------------------------------------------------------------
>
> Key: ARROW-3324
> URL: https://issues.apache.org/jira/browse/ARROW-3324
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Reporter: Wes McKinney
> Priority: Major
> Labels: parquet
> Fix For: 0.12.0
>
>
> See:
> * https://github.com/apache/arrow/issues/2614
> * https://github.com/apache/arrow/issues/2624
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
