prawalpokharel commented on issue #40653:
URL: https://github.com/apache/arrow/issues/40653#issuecomment-6957939388

   I looked at this on current main and pyarrow 26.0.1. As the description 
says, `Close()` doesn't help, because it also reads to the end of the stream, 
with or without threads. Without threads, destroying any scan iterator early 
drains it too. That includes `Head()`, `TakeRows()` and pyarrow's 
`to_batches()`, which all stop reading the `ScanBatches()` iterator early.
   
   On a 4 GiB Parquet dataset (32 files, 4 cores, Linux), read through 
`Scanner.to_reader()` and stopped after 10 batches:
   
   - `reader.close()` took 1.5 to 2.7 s and read another 3.4 to 3.5 GiB (84 to 
88% of the dataset).
   - Dropping a reader made with `use_threads=False` took 1.2 to 2.5 s and read 
3.5 to 3.7 GiB.
   - `dataset.head(5, use_threads=False)` took as long as reading the whole 
dataset without threads.
   - Dropping a threaded reader took under 10 ms, because the `stop_producing` 
deleter in `ScanBatchesUnorderedAsync` already calls `StopProducing()`.
   
   After every cycle the memory pool was back to zero and no dataset file was 
still open, so I saw no leak, only the extra time and I/O.
   
   I'd suggest doing what `PlanReader::Close()` in Acero has done since 
GH-35935: stop the plan, then drain it. `ScanBatches()`, 
`ScanBatchesUnordered()` and `ToRecordBatchReader()` would return an iterator 
that holds a `weak_ptr` to the scan's `ExecPlan` and calls `StopProducing()` 
when destroyed early. The reader's `Close()` would do the same before draining. 
The drain in `SerialIterator`'s destructor stays, since removing it caused 
leaks and hanging tests in #44162. With the plan stopped, the drain ends 
quickly. Reads that were already in flight would then finish in the background 
after `Close()` returns, as they already do when a threaded reader is dropped.
   
   I've opened #55660 with this. It's +95/−23 in scanner.cc and a few doc lines 
in scanner.h, plus tests in scanner_test.cc that fail on main for each of the 
drains above.
   
   pitrou, the issue is still assigned to you from #44162, so let me know if 
you had a different approach in mind.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to