unReaLaneS commented on issue #37139:
URL: https://github.com/apache/arrow/issues/37139#issuecomment-1677160681
Hi, thanks for answering. I checked the memory usage, it seems
pq.read_table().to_batches() loads entire file into memory, is it supposed to
do so, cause it can be misunderstood that to_batches is going to take partial
data. I also tracked locally and on the server, it is fetching entire file and
then, until next file comes it is not calling s3 at all.
Solution that I found for this issue, is to switch to the:
`pq.ParquetFile("bucket_path",
filesystem=self.s3_file_system).iter_batches(batch_size)`, which indeed loads
data in batches into memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
