etseidl commented on code in PR #11438:
URL: https://github.com/apache/arrow-rs/pull/11438#discussion_r4235046972
##########
parquet/src/file/metadata/push_decoder.rs:
##########
@@ -489,6 +470,97 @@ impl ParquetMetaDataPushDecoder {
}
}
+ /// Try to decode the page indexes from the pushed data, returning the
+ /// decoded [`PageIndex`] or an error if not enough data is available.
+ ///
+ /// This method can only be called after [`Self::try_new_with_metadata`],
+ /// which initializes the decoder with previously decoded metadata.
+ ///
+ /// This method is useful when you want to:
+ /// 1. First decode the footer metadata without reading the page indexes
+ /// (using [`PageIndexPolicy::Skip`])
+ /// 2. Later selectively decode a subset of the page indexes using
+ /// [`Self::with_column_index_mask`] or [`Self::with_offset_index_mask`]
+ ///
+ /// When the data can be accessed through a [`ChunkReader`], prefer
+ /// [`ParquetMetaDataReader::read_page_index`] (or
+ /// [`ParquetMetaDataReader::read_page_index_async`] for asynchronous
+ /// I/O), which handles the decoder state machine and I/O requests.
+ ///
+ /// # Returns
+ ///
+ /// * `Ok(DecodeResult::Data(Some(page_index)))` - Successfully decoded the
+ /// requested page indexes based on the configured policies and masks
+ /// * `Ok(DecodeResult::Data(None))` - No page indexes were requested
+ /// (both policies set to [`PageIndexPolicy::Skip`] or masks select
nothing)
+ /// * `Ok(DecodeResult::NeedsData(ranges))` - More data is needed to decode
+ /// the page indexes. Push the requested ranges and call this method
again.
+ /// * `Ok(DecodeResult::Finished)` - The decoder has finished and no more
+ /// data can be decoded
+ /// * `Err(_)` - An error occurred during decoding
+ ///
+ /// [`ParquetMetaDataReader::read_page_index`]:
crate::file::metadata::ParquetMetaDataReader::read_page_index
+ /// [`ParquetMetaDataReader::read_page_index_async`]:
crate::file::metadata::ParquetMetaDataReader::read_page_index_async
+ pub fn try_decode_page_index(&mut self) ->
Result<DecodeResult<Option<PageIndex>>> {
+ match std::mem::replace(&mut self.state, DecodeState::Intermediate) {
+ DecodeState::ReadingPageIndex(metadata) => {
+ Ok(match self.try_decode_page_index_for_metadata(metadata)? {
+ PageIndexDecodeResult::NeedsData(ranges) =>
DecodeResult::NeedsData(ranges),
+ PageIndexDecodeResult::Data { page_index, .. } => {
+ DecodeResult::Data(page_index)
+ }
+ })
+ }
+ DecodeState::Finished => Ok(DecodeResult::Finished),
+ _ => Err(general_err!(
+ "ParquetMetaDataPushDecoder: internal error, invalid state"
+ )),
+ }
+ }
+
+ fn try_decode_page_index_for_metadata(
+ &mut self,
+ metadata: Box<ParquetMetaData>,
+ ) -> Result<PageIndexDecodeResult> {
+ let range = range_for_page_index(
+ &metadata,
+ self.column_index_policy,
+ self.offset_index_policy,
+ &self.column_index_mask,
+ &self.offset_index_mask,
+ );
+
+ let Some(page_index_range) = range else {
+ self.state = DecodeState::Finished;
+ return Ok(PageIndexDecodeResult::Data {
+ metadata,
+ page_index: None,
+ });
+ };
+
+ if !self.buffers.has_range(&page_index_range) {
+ self.state = DecodeState::ReadingPageIndex(metadata);
+ return
Ok(PageIndexDecodeResult::NeedsData(vec![page_index_range]));
+ }
+
+ let buffer = self.get_bytes(&page_index_range)?;
+ let page_index = parse_page_index(
+ &metadata,
+ self.column_index_policy,
+ self.offset_index_policy,
+ &self.column_index_mask,
+ &self.offset_index_mask,
+ &buffer,
+ page_index_range.start,
+ )?;
+
+ self.state = DecodeState::Finished;
Review Comment:
Yeah, I think the model would be create a new decoder to get the state back
to `ReadingPageIndex`.
In my mind it's start with metadata and no page index. When a query comes
in, use this new API to grab the bits you need, store them somewhere, and then
create a new PageIndex (or custom PageIndexProvider), stick that into a cloned
metadata and execute query with that metadata.
New query comes in, figure out the needed indexes, compare that with what's
already been fetched, use this new API to get the new bits, merge those into
the cache, then create new PageIndex and cloned metadata to satisfy query 2.
Rinse repeat.
Again, I think the example in #11475 is my current thinking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
