jiangxt2 commented on code in PR #54926:
URL: https://github.com/apache/arrow/pull/54926#discussion_r4643335836
##########
python/pyarrow/_orc.pyx:
##########
@@ -364,6 +366,53 @@ cdef class ORCReader(_Weakrefable):
return pyarrow_wrap_batch(sp_record_batch)
+ def iter_batches(self, int64_t batch_size, stripes=None, columns=None):
+ cdef:
+ shared_ptr[CRecordBatchReader] batch_reader
+ shared_ptr[CRecordBatch] batch
+ std_vector[c_string] names
+ optional[std_vector[c_string]] selected_names
+ int64_t stripe
+
+ if columns is not None:
+ names = [tobytes(name) for name in columns]
+ selected_names = optional[std_vector[c_string]](names)
+
+ # None selects a whole-file reader. Zero-column projection needs
+ # individual stripe readers to preserve row counts.
+ if stripes is None and (columns is None or len(columns) != 0):
+ stripe_indices = [None]
Review Comment:
I kept the whole-file reader for `stripes=None` when reading all columns or
using a non-empty projection. Each liborc `RowReader` initialization walks the
file's stripe metadata, so creating one per stripe would make that work
quadratic for a full scan. The whole-file path also preserves invalid-column
validation for empty files. I capped native batch capacity on both paths.
##########
cpp/src/arrow/adapters/orc/adapter.cc:
##########
@@ -156,6 +156,12 @@ class OrcStripeReader : public RecordBatchReader {
return Status::OK();
}
+ if (schema_->num_fields() == 0) {
Review Comment:
Thanks for pointing this out. `iter_batches(columns=[])` preserves the row
count, while `ORCFile.read(columns=[])` and `read_stripe(i, columns=[])`
currently return zero rows. I clarified that the column selection syntax
matches `read_stripe`. Since #54921 proposes keeping the existing read APIs'
behavior unchanged, would you be comfortable addressing their zero-column row
counts in a separate follow-up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]