SteNicholas commented on code in PR #1469:
URL: https://github.com/apache/paimon-rust/pull/1469#discussion_r4656859830


##########
crates/paimon/src/arrow/format/parquet.rs:
##########
@@ -88,9 +94,90 @@ impl ParquetFormatReader {
             page_index_enabled: 
core_options.parquet_filter_column_index_enabled()?,
             // Format-table files may be replaced in place.
             metadata_cache_enabled: !core_options.is_format_table(),
+            data_evolution_enabled: core_options.data_evolution_enabled(),
+            metadata: None,
         })
     }
 
+    pub(crate) fn with_metadata(mut self, metadata: ArrowReaderMetadata) -> 
Self {
+        self.metadata = Some(metadata);
+        self
+    }
+
+    fn reader_options(&self, has_predicate: bool, has_selection: bool) -> 
ArrowReaderOptions {
+        let mut options = ArrowReaderOptions::new();
+        if self.page_index_enabled {
+            if has_predicate || (self.data_evolution_enabled && has_selection) 
{

Review Comment:
   Fixed in 
[b6d10803](https://github.com/apache/paimon-rust/commit/b6d10803a153c667a92434148659e054607f95a9).
 Predicate-free sparse reads load only OffsetIndex, with merged-read pruning 
either enabled or disabled. If loading optional column and offset indexes 
together fails, the reader retries offsets independently before falling back to 
the footer. The retry reuses already-fetched metadata bytes; required 
ColumnIndex errors still propagate.
   
   A counting-I/O regression uses 30,000 one-row pages so the indexes extend 
beyond footer prefetch. Valid and corrupt optional ColumnIndexes now produce 
the same row, metadata bytes and column-data bytes as the ordinary 
OffsetIndex-only path. A separate regression verifies that optional ColumnIndex 
failure preserves the valid offsets with one metadata read. [I/O 
regression](https://github.com/apache/paimon-rust/blob/b6d10803a153c667a92434148659e054607f95a9/crates/paimon/src/arrow/format/parquet.rs#L6975)
 ยท [Retry 
regression](https://github.com/apache/paimon-rust/blob/b6d10803a153c667a92434148659e054607f95a9/crates/paimon/src/arrow/format/parquet.rs#L7039)
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to