prawalpokharel commented on issue #40653: URL: https://github.com/apache/arrow/issues/40653#issuecomment-6957939388
I looked at this on current main and pyarrow 26.0.1. As the description says, `Close()` doesn't help, because it also reads to the end of the stream, with or without threads. Without threads, destroying any scan iterator early drains it too. That includes `Head()`, `TakeRows()` and pyarrow's `to_batches()`, which all stop reading the `ScanBatches()` iterator early. On a 4 GiB Parquet dataset (32 files, 4 cores, Linux), read through `Scanner.to_reader()` and stopped after 10 batches: - `reader.close()` took 1.5 to 2.7 s and read another 3.4 to 3.5 GiB (84 to 88% of the dataset). - Dropping a reader made with `use_threads=False` took 1.2 to 2.5 s and read 3.5 to 3.7 GiB. - `dataset.head(5, use_threads=False)` took as long as reading the whole dataset without threads. - Dropping a threaded reader took under 10 ms, because the `stop_producing` deleter in `ScanBatchesUnorderedAsync` already calls `StopProducing()`. After every cycle the memory pool was back to zero and no dataset file was still open, so I saw no leak, only the extra time and I/O. I'd suggest doing what `PlanReader::Close()` in Acero has done since GH-35935: stop the plan, then drain it. `ScanBatches()`, `ScanBatchesUnordered()` and `ToRecordBatchReader()` would return an iterator that holds a `weak_ptr` to the scan's `ExecPlan` and calls `StopProducing()` when destroyed early. The reader's `Close()` would do the same before draining. The drain in `SerialIterator`'s destructor stays, since removing it caused leaks and hanging tests in #44162. With the plan stopped, the drain ends quickly. Reads that were already in flight would then finish in the background after `Close()` returns, as they already do when a threaded reader is dropped. I've opened #55660 with this. It's +95/−23 in scanner.cc and a few doc lines in scanner.h, plus tests in scanner_test.cc that fail on main for each of the drains above. pitrou, the issue is still assigned to you from #44162, so let me know if you had a different approach in mind. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
