rok commented on code in PR #54926:
URL: https://github.com/apache/arrow/pull/54926#discussion_r4631862211


##########
cpp/src/arrow/adapters/orc/adapter.cc:
##########
@@ -156,6 +156,12 @@ class OrcStripeReader : public RecordBatchReader {
       return Status::OK();
     }
 
+    if (schema_->num_fields() == 0) {

Review Comment:
   Preserving the row count is the right behaviour (it matches Parquet), but it 
makes `columns=[]` mean different things across the API. On 26.0.1:
   
   ```python
   >>> f.read(columns=[]).num_rows
   0
   >>> f.read_stripe(0, columns=[]).num_rows
   0
   ```
   
   while `iter_batches(columns=[])` keeps the rows. The docs say column 
selection "follows `read_stripe`", and `test_iter_batches_projection` has to 
special-case `[]` because of this.
   
   Should we apply the same short-circuit in `ReadBatch` so all three agree?



##########
python/pyarrow/_orc.pyx:
##########
@@ -364,6 +366,53 @@ cdef class ORCReader(_Weakrefable):
 
         return pyarrow_wrap_batch(sp_record_batch)
 
+    def iter_batches(self, int64_t batch_size, stripes=None, columns=None):
+        cdef:
+            shared_ptr[CRecordBatchReader] batch_reader
+            shared_ptr[CRecordBatch] batch
+            std_vector[c_string] names
+            optional[std_vector[c_string]] selected_names
+            int64_t stripe
+
+        if columns is not None:
+            names = [tobytes(name) for name in columns]
+            selected_names = optional[std_vector[c_string]](names)
+
+        # None selects a whole-file reader. Zero-column projection needs
+        # individual stripe readers to preserve row counts.
+        if stripes is None and (columns is None or len(columns) != 0):
+            stripe_indices = [None]

Review Comment:
   I think we can drop the whole-file branch and the `None` sentinel, and just 
use `range(self.nstripes())` when `stripes is None`.
   
   liborc batches never cross stripe boundaries, so the output is the same 
either way: a dataset scan (which uses `GetRecordBatchReader`) of a file with 4 
stripes of 16384 rows at `batch_size=65536` yields `[16384, 16384, 16384, 
16384]`.
   
   That removes the sentinel, the `names` / `selected_names` duplication, the 
zero-column special case and the `GetRecordBatchReader` declaration in 
`_orc.pxd`. It also means the `batch_size` clamp suggested in `GetStripeReader` 
covers every path.



##########
python/pyarrow/_orc.pyx:
##########
@@ -364,6 +366,53 @@ cdef class ORCReader(_Weakrefable):
 
         return pyarrow_wrap_batch(sp_record_batch)
 
+    def iter_batches(self, int64_t batch_size, stripes=None, columns=None):
+        cdef:
+            shared_ptr[CRecordBatchReader] batch_reader
+            shared_ptr[CRecordBatch] batch
+            std_vector[c_string] names
+            optional[std_vector[c_string]] selected_names
+            int64_t stripe
+
+        if columns is not None:
+            names = [tobytes(name) for name in columns]
+            selected_names = optional[std_vector[c_string]](names)
+
+        # None selects a whole-file reader. Zero-column projection needs
+        # individual stripe readers to preserve row counts.
+        if stripes is None and (columns is None or len(columns) != 0):
+            stripe_indices = [None]
+        elif stripes is None:
+            stripe_indices = range(self.nstripes())
+        else:
+            stripe_indices = stripes
+
+        try:
+            for index in stripe_indices:
+                if index is None:
+                    with nogil:
+                        batch_reader = GetResultValue(
+                            self.reader.get().GetRecordBatchReader(batch_size, 
names))
+                else:
+                    stripe = index
+                    with nogil:
+                        batch_reader = GetResultValue(
+                            self.reader.get().GetStripeReader(
+                                stripe, batch_size, selected_names))
+                while True:
+                    if self.source_handle.get().closed():

Review Comment:
   I'd remove this check, together with the `source_handle` member and 
`test_iter_batches_closed_source_with_buffered_rows`.
   
   Reading from a closed source already raises as soon as liborc needs I/O 
(`read_stripe` after close raises `OSError` today). This only adds an error for 
rows liborc has already buffered, which is harmless, and no other pyarrow 
reader does it (`ParquetFile.iter_batches` doesn't). It also raises 
`ValueError` where the real I/O path raises `OSError`, which is why 
`test_iter_batches_errors_after_output` has to accept three exception types.
   
   (Agree with your reply to the earlier thread though: `.closed` is already 
required at open, so this isn't a new prerequisite.)



##########
docs/source/python/orc.rst:
##########
@@ -167,6 +167,41 @@ concatenate them into a single table. You can read 
individual stripes with
    two: ["foo","bar","baz"]
    three: [true,false,true]
 
+To consume a file in batches without first materializing an entire stripe as
+Arrow data, use :meth:`~pyarrow.orc.ORCFile.iter_batches`:
+
+.. code-block:: python
+
+   >>> batches = orc_file.iter_batches(batch_size=2, stripes=[0], 
columns=['one'])
+   >>> [batch.num_rows for batch in batches]
+   [2, 1]
+
+Omitting ``stripes`` reads the entire file in physical order. An explicit
+sequence reads stripes in the supplied order, including repetitions; an empty
+sequence reads nothing. Column selection follows ``read_stripe``: use column
+names (including nested paths) or top-level integer indices. Mixing names and
+indices is not supported. ``columns=[]`` returns zero-column batches with the
+correct row counts.
+
+Reading begins when the iterator is consumed. When stopping early, call
+``close()`` to release its reader:
+
+.. code-block:: python
+
+   >>> batches = orc_file.iter_batches(batch_size=2)
+   >>> try:
+   ...     first_batch = next(batches)
+   ... finally:
+   ...     batches.close()
+
+Closing the iterator does not close a file object supplied by the caller.
+Returned batches remain valid. ``batch_size`` bounds the number of output rows,

Review Comment:
   Nit: this paragraph could be shorter. I'd keep "closing the iterator does 
not close the source" and one sentence saying `batch_size` limits rows rather 
than memory, and drop the rest.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to