unReaLaneS commented on issue #37139: URL: https://github.com/apache/arrow/issues/37139#issuecomment-1682123815
It seems, it is a problem with memory leak, when I get batches in to_batches or iter_batches and loop through generator of RecordBatch, it is taking up memory on every iteration and it is not getting freed. Here is the memory graph after loading 198 GB of data(compressed in multiple parquet files) in AWS FargateTask, it is being killed after it runs out of memory. I tried with gc.collect() and every optimization, deleting object, but still it keeps going up, even when I switch to load other file, the memory taken by previous file, it is still allocated.  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
