prawalpokharel opened a new pull request, #55660: URL: https://github.com/apache/arrow/pull/55660
### Rationale for this change Ending a dataset scan early can block until the rest of the dataset has been read (#40653): - `ScannerRecordBatchReader::Close()` reads the rest of the stream, with or without threads. - Without threads, destroying a scan iterator drains it too: the iterator is a `SerialIterator`, whose destructor runs the generator to the end. This affects dropping the reader from `ToRecordBatchReader()`, as well as `Head()`, `TakeRows()` and pyarrow's `to_batches()`, which all stop reading the `ScanBatches()` iterator early. Measured with one pyarrow build, swapping only `libarrow_dataset`, on 4 GiB of Parquet in 32 files with 4 numeric columns (default readahead, 16-core Linux), stopping after 10 batches. Each cell is the median of 10 cycles in 2 processes: the time to close or drop, then the data read after the stop (`rchar` from `/proc/self/io`): | | main | this PR | |---|---|---| | `reader.close()` | 1130 ms, 3.5 GiB | 86 ms, 32 MiB | | `reader.close()`, `use_threads=False` | 970 ms, 3.7 GiB | 58 ms, 159 MiB | | `del reader`, `use_threads=False` | 1003 ms, 3.7 GiB | 53 ms, 143 MiB | | `del` a `to_batches(use_threads=False)` generator | 1122 ms, 3.7 GiB | 58 ms, 143 MiB | | `dataset.head(5, use_threads=False)`, whole call | 1475 ms | 228 ms | With 24 string columns instead, `reader.close()` went from 1199 ms and 3.4 GiB to 76 ms and 127 MiB. Dropping a threaded reader was already fast, and neither it nor full reads changed measurably. In an interleaved A/B over 12 rounds, the median ratio of the time to read everything through `to_reader()` with this PR to the time on main was 0.999 and 1.014 for two schemas (sign-flip permutation test, p = 0.98 and 0.87). ### What changes are included in this PR? `ScanBatches()`, `ScanBatchesUnordered()` and `ToRecordBatchReader()` now return an iterator that holds a `weak_ptr` to the scan's `ExecPlan` and calls `StopProducing()` when it is destroyed before the end. The `SerialIterator` drain is unchanged, so the iterator still waits for the plan to finish, which now happens right after the stop. `ScannerRecordBatchReader::Close()` stops the plan the same way and then reads until the plan has finished, treating `Cancelled` as the end, like `PlanReader::Close()` in Acero (GH-35935). `Close()` also releases the iterator, including on error, so the reader no longer holds the scan once `Close()` returns. The iterator and the reader both hold the plan through a `weak_ptr`, so neither keeps it, or its files, alive after the end of the scan (GH-41771). A scan that is read to the end behaves as before. Not changed: Acero's own `DeclarationToReader()` reader, which still drains when it is destroyed without `Close()` and without threads. ### Are these changes tested? Yes, `TestScanEarlyStop` in `scanner_test.cc`, with and without threads: closing a reader, destroying a reader, destroying a `ScanBatches()` iterator, and closing a reader after reading everything. On main, closing a reader early (in both modes) and destroying a reader or iterator without threads fail, because all 1000 batches are requested. The threaded destroy cases already pass on main. With the change, a serial scan requests 16 batches and a threaded one 8 to 13. On Linux x86_64: - The new tests passed 500 runs in a debug build, 200 runs each under ASAN/UBSAN and TSAN, 300 runs in a one-CPU container and 3200 runs across 64 parallel copies. - The `arrow_dataset` and `arrow_acero` ctest labels pass. - pyarrow's test suite has the same results with and without the change: everything that runs passes except 3 tests that build a Cython or C++ extension, which fail either way because they can't link in this setup. ### Are there any user-facing changes? - Closing or dropping a scanner's reader early, dropping a `ScanBatches()` or `ScanBatchesUnordered()` iterator early, `Head()` and `TakeRows()` no longer read the rest of the dataset. - Reads that were already in flight when the scan stopped can finish in the background after `Close()` returns. In the runs above, right after an early close or drop without threads, up to 4 dataset files were still open and medians of 391 to 643 MiB still allocated, all released within about 0.2 s (medians of 21 to 107 ms). Dropping a threaded reader on main already behaves this way. `Close()` used to return with nothing in flight only because it had read everything. The behaviour is documented in `scanner.h`. - `Close()` no longer reports errors from data it doesn't read. If the scan was cancelled, `Close()` returns OK instead of `Cancelled`. ### Was AI used for this PR? In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR. **PR code and description written by:** - [x] Human - [x] AI **Reviewed before submission by:** - [x] Human - [ ] AI - [ ] Not reviewed I used an AI coding assistant to help write the change and the tests and to run the measurements. I've reviewed all of it and will handle the review myself. * GitHub Issue: #40653 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
