etseidl commented on code in PR #10719:
URL: https://github.com/apache/arrow-rs/pull/10719#discussion_r3817623397
##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -1313,12 +1313,11 @@ impl<T: ChunkReader + 'static> ReaderPageIterator<T> {
fn next_page_reader(&self, rg_idx: usize) ->
Result<SerializedPageReader<T>> {
let rg = self.metadata.row_group(rg_idx);
let column_chunk_metadata = rg.column(self.column_idx);
- let offset_index = self.metadata.offset_index();
- // `offset_index` may not exist and `i[rg_idx]` will be empty.
- // To avoid `i[rg_idx][self.column_idx`] panic, we need to filter out
empty `i[rg_idx]`.
- let page_locations = offset_index
- .filter(|i| !i[rg_idx].is_empty())
- .map(|i| i[rg_idx][self.column_idx].page_locations.clone());
+ let page_locations = self
+ .metadata
+ .page_index()
+ .map(|i| i.page_locations(rg_idx, self.column_idx).cloned())
Review Comment:
> Where @zhuqi-lucas and others have been looking for a way to load some but
not all page indexes (or load them on demand, from a cache, etc).
I think the new form will help with that...I envision a builder that starts
out with empty vecs sized with num_row_groups and num_columns. Individual cells
can then be populated based on what it needed for a given use. We can pop back
and forth between page index and page index builder if need be.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
