jiangxt2 commented on code in PR #54926:
URL: https://github.com/apache/arrow/pull/54926#discussion_r4643338294


##########
python/pyarrow/_orc.pyx:
##########
@@ -364,6 +366,53 @@ cdef class ORCReader(_Weakrefable):
 
         return pyarrow_wrap_batch(sp_record_batch)
 
+    def iter_batches(self, int64_t batch_size, stripes=None, columns=None):
+        cdef:
+            shared_ptr[CRecordBatchReader] batch_reader
+            shared_ptr[CRecordBatch] batch
+            std_vector[c_string] names
+            optional[std_vector[c_string]] selected_names
+            int64_t stripe
+
+        if columns is not None:
+            names = [tobytes(name) for name in columns]
+            selected_names = optional[std_vector[c_string]](names)
+
+        # None selects a whole-file reader. Zero-column projection needs
+        # individual stripe readers to preserve row counts.
+        if stripes is None and (columns is None or len(columns) != 0):
+            stripe_indices = [None]
+        elif stripes is None:
+            stripe_indices = range(self.nstripes())
+        else:
+            stripe_indices = stripes
+
+        try:
+            for index in stripe_indices:
+                if index is None:
+                    with nogil:
+                        batch_reader = GetResultValue(
+                            self.reader.get().GetRecordBatchReader(batch_size, 
names))
+                else:
+                    stripe = index
+                    with nogil:
+                        batch_reader = GetResultValue(
+                            self.reader.get().GetStripeReader(
+                                stripe, batch_size, selected_names))
+                while True:
+                    if self.source_handle.get().closed():

Review Comment:
   Removed the per-batch closed-source check, `source_handle` and the 
buffered-rows test. The ownership and early-close tests remain. The 
read-failure test now covers closing the source after the first batch and then 
attempting to read a later stripe on the explicit-stripe path.
   



##########
docs/source/python/orc.rst:
##########
@@ -167,6 +167,41 @@ concatenate them into a single table. You can read 
individual stripes with
    two: ["foo","bar","baz"]
    three: [true,false,true]
 
+To consume a file in batches without first materializing an entire stripe as
+Arrow data, use :meth:`~pyarrow.orc.ORCFile.iter_batches`:
+
+.. code-block:: python
+
+   >>> batches = orc_file.iter_batches(batch_size=2, stripes=[0], 
columns=['one'])
+   >>> [batch.num_rows for batch in batches]
+   [2, 1]
+
+Omitting ``stripes`` reads the entire file in physical order. An explicit
+sequence reads stripes in the supplied order, including repetitions; an empty
+sequence reads nothing. Column selection follows ``read_stripe``: use column
+names (including nested paths) or top-level integer indices. Mixing names and
+indices is not supported. ``columns=[]`` returns zero-column batches with the
+correct row counts.
+
+Reading begins when the iterator is consumed. When stopping early, call
+``close()`` to release its reader:
+
+.. code-block:: python
+
+   >>> batches = orc_file.iter_batches(batch_size=2)
+   >>> try:
+   ...     first_batch = next(batches)
+   ... finally:
+   ...     batches.close()
+
+Closing the iterator does not close a file object supplied by the caller.
+Returned batches remain valid. ``batch_size`` bounds the number of output rows,

Review Comment:
   Shortened the paragraph to retain the iterator and source ownership 
behavior, and the distinction between the batch row limit and total memory.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to