etseidl commented on code in PR #11157:
URL: https://github.com/apache/arrow-rs/pull/11157#discussion_r4088308132
##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -774,6 +776,32 @@ impl ArrowReaderOptions {
self
}
+ /// Sets the [`ColumnChunkMask`] for the Parquet [ColumnIndex] structure.
+ ///
+ /// The column index can be costly to decode and store, especially when it
is needed
+ /// only for a subset of row groups or columns (such as when filtering by
a predicate
+ /// on a single column). Providing a [`ColumnChunkMask`] can greatly
decrease
+ /// the time needed to decode this metadata.
+ ///
+ /// [ColumnIndex]:
https://github.com/apache/parquet-format/blob/master/PageIndex.md
+ pub fn with_column_index_mask(mut self, mask: ColumnChunkMask) -> Self {
+ self.column_index_mask = mask;
+ self
+ }
+
+ /// Sets the [`ColumnChunkMask`] for the Parquet [OffsetIndex] structure.
+ ///
+ /// The offset index can be costly to decode and store, especially when it
is needed
+ /// only for a subset of row groups or columns (such as when projecting a
small subset
+ /// of columns). Providing a [`ColumnChunkMask`] can greatly decrease
+ /// the time needed to decode this metadata.
+ ///
+ /// [OffsetIndex]:
https://github.com/apache/parquet-format/blob/master/PageIndex.md
+ pub fn with_offset_index_mask(mut self, mask: ColumnChunkMask) -> Self {
+ self.offset_index_mask = mask;
+ self
+ }
Review Comment:
I think this one's getting a little pedantic 😅. I mean, I think users can
figure out that a function that explicitly states that it won't load the page
index won't be affected by options controlling the loading of the page index.
Fair point re noting that you'll want the offset index for the predicate
columns as well as the projected columns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
