GujaLomsadze opened a new pull request, #50646:
URL: https://github.com/apache/arrow/pull/50646
### Rationale for this change
`RecordBatchFileReader::CountRows` has existed in Arrow C++
(`cpp/src/arrow/ipc/reader.h`) but
was never bound in Python, so the only way to get the total number of rows
of an IPC file was:
```python
num_rows = sum(reader.get_batch(i).num_rows for i in
range(reader.num_record_batches))
```
That deserializes every record batch just to read its length, which is
wasteful and becomes
expensive on remote filesystems.
### What changes are included in this PR?
Adds `RecordBatchFileReader.count_rows()`:
```python
with pa.ipc.open_file(source) as reader:
reader.count_rows()
```
Three changes:
* `python/pyarrow/includes/libarrow.pxd`: declare `CResult[int64_t]
CountRows()` on
`CRecordBatchFileReader`, which was the missing piece.
* `python/pyarrow/ipc.pxi`: add `count_rows()` to `_RecordBatchFileReader`,
released GIL around
the call, with the same closed reader guard used by the existing `stats`
property
* `python/pyarrow/tests/test_ipc.py`: tests.
On the naming: `count_rows()` follows the C++ method and is consistent with
the existing
`count_rows()` on `Dataset`, `Scanner` and `Fragment`.
To be precise about the benefit, since the issue describes it as reading the
count from the
metadata: the C++ implementation still walks every block, but reads only
each record batch's
flatbuffer message header to pick up its length, and never touches the data
buffers. So this is
a reduction in bytes read rather than in the number of reads, and the gain
shows up on remote
filesystems and on files with large batches rather than in a local in memory
benchmark.
This is only added to the file reader. The stream reader has no footer and
cannot count rows
without consuming the stream.
### Are these changes tested?
Yes, two tests in `python/pyarrow/tests/test_ipc.py`:
* `test_file_count_rows`: count matches the sum of the written batch
lengths, and counting does
not consume the reader (count, `read_all()`, count again).
* `test_file_count_rows_no_batches`: a file with a schema but no batches
counts 0.
Locally `test_ipc.py` passes (72 tests) and `test_feather.py` passes (83
passed, 8 skipped,
1 xfailed), the latter because the feather reader sits on the same file
reader. The docstring
example was run and produces the output shown.
### Are there any user-facing changes?
Yes, a new public method `RecordBatchFileReader.count_rows()`. No existing
behaviour changes.
### AI usage disclosure
I used Claude Code to locate the unbound C++ method and the place where the
declaration was
missing, and to draft the binding, the docstring and the tests. I reviewed
the result, rebuilt
PyArrow locally, ran the test suites quoted above, and checked the C++
implementation of
`CountRows` myself to confirm what it actually does before describing the
benefit here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
