StuartHadfield opened a new issue, #36303:
URL: https://github.com/apache/arrow/issues/36303
### Describe the usage question you have. Please include as many useful
details as possible.
First off, thanks for all the work on Arrow / PyArrow.
I'm using the `datasets` module from PyArrow in a production context - and I
am perplexed by how the memory usage is managed internally.
The test case I've been working with is thusly:
- A 30GB input CSV
- Partitioned in Hive Style Partitioning
- Written to S3 in Parquet Format, using the `write_dataset` function.
- Executed in a Docker Container with <2GB of memory
My overall process looks something like:
```py
def get_rows():
rows = parse_file(my_file) # A magic function that reads rows from a CSV
file
for i, row in enumerate(rows):
if i % 1_000_000:
print(f"Yielding row number {i}")
yield row
def preprocess_rows(rows):
for row in rows:
# do some processing on individual values in the rows, e.g.
typecasting values
yield row
def generate_record_batches(rows: Iterable[Dict[Any, Any]], schema):
for chunk in chunked(rows, 1024):
yield pa.RecordBatch.from_pylist(chunk, schema=schema)
def write_to_parquet(rows, schema, ...):
processed_rows = preprocess_rows(rows)
ds.write_dataset(
data=generate_record_batches(processed_rows, schema)
max_rows_per_file=500_000,
max_rows_per_group=500_000,
# various other kwargs, e.g. compression etc.
)
```
With that in mind, I've been monitoring:
1. The memory consumption of my docker container
2. The data that's been written to S3
And I've seen the following:
- The memory consumption approached 1.1GB, and has since levelled off. It
moves a few tens of MB up and down as the write process goes on
- After having processed ~15 million rows, PyArrow has written the prefixes
(i.e. folders) to S3 that the data will live in, but _no actual data_ has been
written - that is to say no parquet files are present in my bucket.
This seems like total voodoo to me. My assumption would be that once the
writer has accumulated 500,000 rows, it'll flush them. But, of course, this
means it has to have 500,000 rows _per partition_. In my case, I have 200
partitions, so I will need to have read 100,000,000 rows before it'll flush
(assuming rows are evenly distributed between partitions)?
If that is true, how does memory consumption not increase linearly with rows
read? If my dataset is 30GB @ 70 million rows, and I've read 15 million, it
stands to reason that ~2 million rows should be about 1GB, and so I should've
read 7GB of data. What magic is happening under the hood?
The next question I have is related to the discovery of the partition values
from the dataset. How does PyArrow know these up front, such that it may create
the folder structure on S3 immediately?
Thanks for taking the time.
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
