jiangxt2 opened a new pull request, #54926: URL: https://github.com/apache/arrow/pull/54926
### Rationale for this change `ORCFile.read_stripe()` materializes an entire stripe as one record batch. Dataset already scans ORC in batches; this adds native batching with explicit stripe selection to `ORCFile`, supporting incremental consumption and early termination. Closes #54921. ### What changes are included in this PR? - Add `ORCFile.iter_batches(batch_size=65536, stripes=None, columns=None)` with column projection. - Add stateless C++ `GetStripeReader` overloads that decode directly into batches without changing the reader's Seek position. - Handle zero-column row counts and iterator lifetimes, with API documentation and regression tests. ### Are these changes tested? Validated on macOS arm64 with matching Release C++/Python builds, Python 3.13.7 and ORC 2.2.1. ```bash cpp/build/dev/release/arrow-orc-adapter-test ``` 44 passed; one existing timezone-database test restricted to ORC < 2.0 was skipped. From `python/`: ```bash python -m pytest -q -ra --enable-orc --enable-dataset --enable-pandas pyarrow/tests/test_orc.py pyarrow/tests/test_dataset.py::test_orc_format pyarrow/tests/test_dataset.py::test_orc_scan_options pyarrow/tests/test_dataset.py::test_orc_writer_not_implemented_for_dataset ``` 232 passed. Coverage includes five codecs, stripe order and duplicates, batch limits, nested/zero-column projections, invalid arguments, interleaved iterators, source lifetimes and read failures. All five codec variants of the lifecycle test also passed its close-before-first-read assertions. The C++ memory regression uses four int64 columns and batches of 256 rows. For single stripes of 262144 and 1048576 rows, whole-stripe Arrow allocation peaks were 16.125 and 64.5 MiB; streaming peaked at 16.25 KiB for both sizes. NONE/ZLIB range tests verify that consuming the first batch and closing early does not read unselected stripe data. Changed-file format/lint checks and `git diff --check` passed. ### Are there any user-facing changes? Yes. The new API yields `RecordBatch` objects. `stripes=None` reads all stripes; explicit sequences preserve order and duplicates, and empty sequences yield no batches. `columns=[]` preserves row counts. Closing the iterator releases its reader without closing the input; returned batches remain valid. `batch_size` limits output rows, not total process memory. ORC buffers and dictionaries require additional memory, and small batches can reduce throughput. Reading the same stripes does not inherently reduce I/O. ### Was AI used for this PR? **PR code and description written by:** - [x] Human - [x] AI **Reviewed before submission by:** - [x] Human - [x] AI - [ ] Not reviewed The human contributor directed the design and reviewed the proposal and PR text. OpenAI Codex assisted with code, tests, documentation and static review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
