[ https://issues.apache.org/jira/browse/ARROW-6910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16968654#comment-16968654 ]
V Luong edited comment on ARROW-6910 at 11/6/19 8:08 PM: --------------------------------------------------------- [~apitrou] [~wesm] [~jorisvandenbossche] I'm re-testing this issue using the newly-released 0.15.1, with the following code, in an interactive Python 3.7 shell: ----- from pyarrow.parquet import read_table import os from tqdm import tqdm PARQUET_S3_PATH = 's3://public-parquet-test-data/big.snappy.parquet' PARQUET_HTTP_PATH = 'http://public-parquet-test-data.s3.amazonaws.com/big.snappy.parquet' PARQUET_TMP_PATH = '/tmp/big.snappy.parquet' os.system('wget --output-document={} {}'.format(PARQUET_TMP_PATH, PARQUET_HTTP_PATH)) for _ in tqdm(range(10)): read_table( source=PARQUET_TMP_PATH, columns=None, use_threads=False, metadata=None, use_pandas_metadata=False, memory_map=False, filesystem=None, filters=None) ----- I observe the following mysterious behavior: - If I don't do anything after the above For loop, the program still occupies 8-10GB of memory and does not release it. I keep it at this idle state for a good 10-15 minutes and confirm that memory is still occupied. - Then, I try to do something random, like "import pyarrow; print(pyarrow.__version__)" in the interactive shell, and then the memory is immediately released. This behavior remains unintuitive to me, and it seems users still don't have a firm control on the memory used by PyArrow. Each read_table(...) call does not seem memory-neutral by default as of 0.15.1 yet. This means long-running iterative programs, especially ML training involving repeated loading up these files, will inevitably OOM. was (Author: mbalearnstocode): [~apitrou] [~wesm] [~jorisvandenbossche] I'm re-testing this issue using the newly-released 0.15.1, with the following code, in an interactive Python 3.7 shell: ----- from pyarrow.parquet import read_table import os from tqdm import tqdm PARQUET_S3_PATH = 's3://public-parquet-test-data/big.snappy.parquet' PARQUET_HTTP_PATH = 'http://public-parquet-test-data.s3.amazonaws.com/big.snappy.parquet' PARQUET_TMP_PATH = '/tmp/big.snappy.parquet' os.system('wget --output-document={} {}'.format(PARQUET_TMP_PATH, PARQUET_HTTP_PATH)) for _ in tqdm(range(10)): read_table( source=PARQUET_TMP_PATH, columns=None, use_threads=False, metadata=None, use_pandas_metadata=False, memory_map=False, filesystem=None, filters=None) ------- I observe the following mysterious behavior: - If I don't do anything after the above For loop, the program still occupies 8-10GB of memory and does not release it. I keep it at this idle state for a good 10-15 minutes and confirm that memory is still occupied. - Then, I try to do something random, like "import pyarrow; print(pyarrow.__version__)" in the interactive shell, and then the memory is immediately released. This behavior remains unintuitive to me, and it seems users still don't have a firm control on the memory used by PyArrow. Each read_table(...) call does not seem memory-neutral by default as of 0.15.1 yet. This means long-running iterative programs, especially ML training involving repeated loading up these files, will inevitably OOM. > [Python] pyarrow.parquet.read_table(...) takes up lots of memory which is not > released until program exits > ---------------------------------------------------------------------------------------------------------- > > Key: ARROW-6910 > URL: https://issues.apache.org/jira/browse/ARROW-6910 > Project: Apache Arrow > Issue Type: Bug > Components: C++, Python > Affects Versions: 0.15.0 > Reporter: V Luong > Assignee: Wes McKinney > Priority: Critical > Labels: pull-request-available > Fix For: 1.0.0, 0.15.1 > > Attachments: arrow6910.png > > Time Spent: 4h 50m > Remaining Estimate: 0h > > I realize that when I read up a lot of Parquet files using > pyarrow.parquet.read_table(...), my program's memory usage becomes very > bloated, although I don't keep the table objects after converting them to > Pandas DFs. > You can try this in an interactive Python shell to reproduce this problem: > ```{python} > from tqdm import tqdm > from pyarrow.parquet import read_table > PATH = '/tmp/big.snappy.parquet' > for _ in tqdm(range(10)): > read_table(PATH, use_threads=False, memory_map=False) > (note that I'm not assigning the read_table(...) result to anything, so > I'm not creating any new objects at all) > ``` > During the For loop above, if you view the memory usage (e.g. using htop > program), you'll see that it keeps creeping up. Either the program crashes > during the 10 iterations, or if the 10 iterations complete, the program will > still occupy a huge amount of memory, although no objects are kept. That > memory is only released when you exit() from Python. > This problem means that my compute jobs using PyArrow currently need to use > bigger server instances than I think is necessary, which translates to > significant extra cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)