chriss1245 commented on issue #37139:
URL: https://github.com/apache/arrow/issues/37139#issuecomment-1763451260
Something similar happens in my case. I am using iter_batches from a
ParquetFile in order to create a generator for tensorflow. The loop is quite
simple.
```python
for batch in data.iter_batches(columns=columns, batch_size=batch_size,
use_threads=False):
batch_df = batch.to_pandas()
yield {
col: tf.convert_to_tensor(batch_df[col].values)
for col in batch_df.columns
}
```
I created a RAM monitoring callback which monitors the ram after each batch.
And I have seen the same behavior: the ram usage increases linearly. I tried
different variations but the pattern is the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
