[
https://issues.apache.org/jira/browse/ARROW-18156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17624593#comment-17624593
]
Weston Pace edited comment on ARROW-18156 at 10/26/22 4:38 PM:
---------------------------------------------------------------
{quote}
What else could affect behavior here? Python version? How the pyarrow package
was installed? I'm using Python 3.8.3.
{quote}
Given this is the system allocator the OS and glibc version might be more
significant than the python version. This could be a pip vs. conda thing also.
I will try and test out pip. The dataset version of scanning will generate a
lot of temporary allocations (probably too many but that is another story). It
seems the allocator is not releasing the RAM.
For reference, here are the results I get. I'm using python 3.9.7 with Arrow
4.0.0 from conda forge.
{noformat}
mode=dataset allocator=jemalloc
jemalloc
Runs: 10
After run 0: RSS = 8.16 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 1: RSS = 9.46 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 2: RSS = 9.47 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 3: RSS = 9.46 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 4: RSS = 9.46 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 5: RSS = 9.47 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 6: RSS = 9.46 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 7: RSS = 9.46 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 8: RSS = 9.46 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 9: RSS = 9.46 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
{noformat}
{noformat}
mode=ParquetFile.read() allocator=jemalloc
jemalloc
Runs: 10
After run 0: RSS = 6.85 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 1: RSS = 8.15 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 2: RSS = 8.16 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 3: RSS = 8.16 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 4: RSS = 8.16 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 5: RSS = 8.16 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 6: RSS = 8.16 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 7: RSS = 8.16 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 8: RSS = 8.16 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 9: RSS = 8.16 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
{noformat}
{noformat}
mode=dataset allocator=jemalloc
system
Runs: 10
After run 0: RSS = 8.16 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 1: RSS = 9.47 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 2: RSS = 9.47 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 3: RSS = 9.47 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 4: RSS = 9.47 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 5: RSS = 9.47 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 6: RSS = 9.48 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 7: RSS = 9.48 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 8: RSS = 9.48 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 9: RSS = 9.48 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
{noformat}
{noformat}
mode=ParquetFile.read() allocator=system
system
Runs: 10
After run 0: RSS = 6.84 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 1: RSS = 8.15 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 2: RSS = 8.15 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 3: RSS = 8.15 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 4: RSS = 8.15 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 5: RSS = 8.15 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 6: RSS = 8.15 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 7: RSS = 8.16 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 8: RSS = 8.15 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 9: RSS = 8.16 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
{noformat}
Note, I am working around the tracking bug [~jorisvandenbossche] mentioned by
setting the default pool using an environment variable.
was (Author: westonpace):
{quote}
What else could affect behavior here? Python version? How the pyarrow package
was installed? I'm using Python 3.8.3.
{quote}
Given this is the system allocator the OS and glibc version might be more
significant than the python version. This could be a pip vs. conda thing also.
I will try and test out pip. The dataset version of scanning will generate a
lot of temporary allocations (probably too many but that is another story). It
seems the allocator is not releasing the RAM.
For reference, here are the results I get. I'm using python 3.9.7 with Arrow
4.0.0 from conda forge.
{{noformat}}
mode=dataset allocator=jemalloc
jemalloc
Runs: 10
After run 0: RSS = 8.16 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 1: RSS = 9.46 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 2: RSS = 9.47 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 3: RSS = 9.46 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 4: RSS = 9.46 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 5: RSS = 9.47 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 6: RSS = 9.46 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 7: RSS = 9.46 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 8: RSS = 9.46 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 9: RSS = 9.46 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
{{noformat}}
{{noformat}}
mode=ParquetFile.read() allocator=jemalloc
jemalloc
Runs: 10
After run 0: RSS = 6.85 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 1: RSS = 8.15 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 2: RSS = 8.16 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 3: RSS = 8.16 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 4: RSS = 8.16 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 5: RSS = 8.16 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 6: RSS = 8.16 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 7: RSS = 8.16 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 8: RSS = 8.16 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 9: RSS = 8.16 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
{{noformat}}
{{noformat}}
mode=dataset allocator=jemalloc
system
Runs: 10
After run 0: RSS = 8.16 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 1: RSS = 9.47 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 2: RSS = 9.47 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 3: RSS = 9.47 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 4: RSS = 9.47 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 5: RSS = 9.47 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 6: RSS = 9.48 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 7: RSS = 9.48 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 8: RSS = 9.48 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 9: RSS = 9.48 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
{{noformat}}
{{noformat}}
mode=ParquetFile.read() allocator=system
system
Runs: 10
After run 0: RSS = 6.84 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 1: RSS = 8.15 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 2: RSS = 8.15 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 3: RSS = 8.15 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 4: RSS = 8.15 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 5: RSS = 8.15 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 6: RSS = 8.15 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 7: RSS = 8.16 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 8: RSS = 8.15 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
After run 9: RSS = 8.16 GB, PyArrow Allocated Bytes = 5.94 GB Table Nbytes =
4.63 GB
{{noformat}}
Note, I am working around the tracking bug [~jorisvandenbossche] mentioned by
setting the default pool using an environment variable.
> [Python/C++] High memory usage/potential leak when reading parquet using
> Dataset API
> ------------------------------------------------------------------------------------
>
> Key: ARROW-18156
> URL: https://issues.apache.org/jira/browse/ARROW-18156
> Project: Apache Arrow
> Issue Type: Bug
> Components: Parquet
> Affects Versions: 4.0.1
> Reporter: Norbert
> Priority: Major
>
> Hi,
> I have a 2.35 GB DataFrame (1.17 GB on-disk size) which I'm loading using the
> following snippet:
>
> {code:java}
> import os
> import pyarrow
> import pyarrow.dataset as ds
> from importlib_metadata import version
> from psutil import Process
> import pyarrow.parquet as pq
> def format_bytes(num_bytes: int):
> return f"{num_bytes / 1024 / 1024 / 1024:.2f} GB"
>
> def main():
> print(version("pyarrow"))
> print(pyarrow.default_memory_pool().backend_name)
> process = Process(os.getpid())
> runs = 10
> print(f"Runs: {runs}")
> for i in range(runs):
> dataset = ds.dataset("df.pq")
> table = dataset.to_table()
> df = table.to_pandas()
> print(f"After run {i}: RSS =
> {format_bytes(process.memory_info().rss)}, PyArrow Allocated Bytes =
> {format_bytes(pyarrow.total_allocated_bytes())}")
> {code}
>
>
> On PyArrow v4.0.1 the output is as follows:
> {code:java}
> 4.0.1
> system
> Runs: 10
> After run 0: RSS = 7.59 GB, PyArrow Allocated Bytes = 6.09 GB
> After run 1: RSS = 13.36 GB, PyArrow Allocated Bytes = 6.09 GB
> After run 2: RSS = 14.74 GB, PyArrow Allocated Bytes = 6.09 GB
> After run 3: RSS = 15.78 GB, PyArrow Allocated Bytes = 6.09 GB
> After run 4: RSS = 18.36 GB, PyArrow Allocated Bytes = 6.09 GB
> After run 5: RSS = 19.69 GB, PyArrow Allocated Bytes = 6.09 GB
> After run 6: RSS = 21.21 GB, PyArrow Allocated Bytes = 6.09 GB
> After run 7: RSS = 21.52 GB, PyArrow Allocated Bytes = 6.09 GB
> After run 8: RSS = 21.49 GB, PyArrow Allocated Bytes = 6.09 GB
> After run 9: RSS = 21.72 GB, PyArrow Allocated Bytes = 6.09 GB
> After run 10: RSS = 20.95 GB, PyArrow Allocated Bytes = 6.09 GB{code}
> If I replace ds.dataset("df.pq").to_table() with
> pq.ParquetFile("df.pq").read(), the output is:
> {code:java}
> 4.0.1
> system
> Runs: 10
> After run 0: RSS = 2.38 GB, PyArrow Allocated Bytes = 1.34 GB
> After run 1: RSS = 2.49 GB, PyArrow Allocated Bytes = 1.34 GB
> After run 2: RSS = 2.50 GB, PyArrow Allocated Bytes = 1.34 GB
> After run 3: RSS = 2.53 GB, PyArrow Allocated Bytes = 1.34 GB
> After run 4: RSS = 2.53 GB, PyArrow Allocated Bytes = 1.34 GB
> After run 5: RSS = 2.56 GB, PyArrow Allocated Bytes = 1.34 GB
> After run 6: RSS = 2.53 GB, PyArrow Allocated Bytes = 1.34 GB
> After run 7: RSS = 2.51 GB, PyArrow Allocated Bytes = 1.34 GB
> After run 8: RSS = 2.48 GB, PyArrow Allocated Bytes = 1.34 GB
> After run 9: RSS = 2.51 GB, PyArrow Allocated Bytes = 1.34 GB
> After run 10: RSS = 2.51 GB, PyArrow Allocated Bytes = 1.34 GB{code}
> The usage profile of the older non-dataset API is much lower - it matches the
> size of the dataframe much closer. It also seems like in the former example,
> there is a memory leak? I thought that the increase in RSS was just due to
> PyArrow's usage of jemalloc, but I seem to be using the system allocator here.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
