aucahuasi commented on code in PR #14226:
URL: https://github.com/apache/arrow/pull/14226#discussion_r995516045
##########
cpp/src/parquet/arrow/reader.cc:
##########
@@ -1168,19 +1168,26 @@
FileReaderImpl::GetRecordBatchGenerator(std::shared_ptr<FileReader> reader,
if (rows_to_readahead < 0) {
return Status::Invalid("rows_to_readahead must be > 0");
}
- if (reader_properties_.pre_buffer()) {
- BEGIN_PARQUET_CATCH_EXCEPTIONS
- reader_->PreBuffer(row_group_indices, column_indices,
reader_properties_.io_context(),
- reader_properties_.cache_options());
- END_PARQUET_CATCH_EXCEPTIONS
- }
- ::arrow::AsyncGenerator<RowGroupGenerator::RecordBatchGenerator>
row_group_generator =
-
RowGroupGenerator(::arrow::internal::checked_pointer_cast<FileReaderImpl>(reader),
- cpu_executor, row_group_indices, column_indices,
- rows_to_readahead);
+ std::vector<::arrow::AsyncGenerator<std::shared_ptr<::arrow::RecordBatch>>>
vec;
+ for (const int& row_group_index: row_group_indices) {
+ if (reader_properties_.pre_buffer()) {
Review Comment:
You are right Weston, doesn't work.
It seems we need to wait to finish the prebuffering process (for each
parquet row group) before we read each row group; I sent new changes, the
relevant part about this is in `RowGroupGenerator::FetchNext` (there is a
future::wait there)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
