JingsongLi commented on code in PR #1469:
URL: https://github.com/apache/paimon-rust/pull/1469#discussion_r4656376210


##########
crates/paimon/src/arrow/format/parquet.rs:
##########
@@ -88,9 +94,90 @@ impl ParquetFormatReader {
             page_index_enabled: 
core_options.parquet_filter_column_index_enabled()?,
             // Format-table files may be replaced in place.
             metadata_cache_enabled: !core_options.is_format_table(),
+            data_evolution_enabled: core_options.data_evolution_enabled(),
+            metadata: None,
         })
     }
 
+    pub(crate) fn with_metadata(mut self, metadata: ArrowReaderMetadata) -> 
Self {
+        self.metadata = Some(metadata);
+        self
+    }
+
+    fn reader_options(&self, has_predicate: bool, has_selection: bool) -> 
ArrowReaderOptions {
+        let mut options = ArrowReaderOptions::new();
+        if self.page_index_enabled {
+            if has_predicate || (self.data_evolution_enabled && has_selection) 
{

Review Comment:
   **[P2] Preserve the OffsetIndex-only path for predicate-free sparse reads**
   
   This also loads the whole file's ColumnIndex for an explicit row-ID read or 
an unfiltered LIMIT, even with 
`data-evolution.merged-read.stats-pushdown.enabled=false`. With a valid 
OffsetIndex and an undecodable optional ColumnIndex, the footer-only retry 
drops the usable OffsetIndex too. A local one-row, one-column read fetched 167 
bytes of column data with the valid indexes and 3,312 bytes after damaging only 
the ColumnIndex. Both returned the same row. Wide cold reads can also pay for 
large valid ColumnIndexes that never participate in pruning.
   
   Keep OffsetIndex-only loading for existing predicate-free selections and 
scope the extra validation to merged reads that need it. If optional 
ColumnIndex loading fails, retry valid offsets independently before falling 
back to full chunks.



##########
crates/paimon/src/arrow/format/parquet.rs:
##########
@@ -2807,51 +2834,96 @@ fn build_predicate_page_selection(
         });
     }
 
-    Ok(combined)
+    combined
 }
 
-/// Collect the file-field indices referenced by leaf predicates (recursing
-/// through compound nodes). Indices are already file-level (see caller).
-fn collect_leaf_field_indices(predicate: &Predicate, out: &mut Vec<usize>) {
-    match predicate {
-        Predicate::NestedLeaf { index, .. } => out.push(*index),
-        Predicate::Leaf { index, .. } => out.push(*index),
-        Predicate::And(children) | Predicate::Or(children) => {
-            for child in children {
-                collect_leaf_field_indices(child, out);
-            }
-        }
-        Predicate::Not(child) => collect_leaf_field_indices(child, out),
-        Predicate::AlwaysTrue | Predicate::AlwaysFalse => {}
-    }
+fn offset_index_valid(metadata: &ParquetMetaData, file_size: u64) -> bool {
+    let Some(groups) = metadata.offset_index() else {
+        return true;
+    };
+    groups.len() == metadata.num_row_groups()
+        && groups.iter().enumerate().all(|(group_index, offsets)| {
+            let group = metadata.row_group(group_index);
+            let stats = metadata
+                .column_index()
+                .and_then(|index| index.get(group_index));
+            offsets.len() == group.num_columns()
+                && offsets.iter().enumerate().all(|(column_index, offset)| {
+                    page_locations_valid(
+                        offset.page_locations(),
+                        group.num_rows(),
+                        group.column(column_index),
+                        file_size,
+                        stats.and_then(|columns| columns.get(column_index)),
+                    )
+                })
+        })
 }
 
-/// Validate that an OffsetIndex's page row boundaries are well-formed for a 
row
-/// group of `rg_rows` rows: the first page starts at row 0, `first_row_index`
-/// is strictly increasing, and every value stays within `[0, rg_rows]`.
-///
-/// Malformed metadata (negative, non-monotonic, or out-of-range boundaries)
-/// would otherwise produce invalid row ranges — huge values from `i64 as 
usize`
-/// underflow, or ranges past the row group — that panic
-/// `RowSelection::from_consecutive_ranges`. Callers fail open when this 
returns
-/// `false`.
-fn page_boundaries_valid(
+fn page_locations_valid(
     pages: &[parquet::file::page_index::offset_index::PageLocation],
-    rg_rows: usize,
+    row_count: i64,
+    column: &ColumnChunkMetaData,
+    file_size: u64,
+    column_index: Option<&ColumnIndexMetaData>,
 ) -> bool {
-    let rg_rows = rg_rows as i64;
-    let mut prev: i64 = -1;
-    for (page_idx, page) in pages.iter().enumerate() {
-        let first_row = page.first_row_index;
-        if page_idx == 0 && first_row != 0 {
-            return false;
-        }
-        if first_row <= prev || first_row > rg_rows {
-            return false;
-        }
-        prev = first_row;
+    let column_index = column_index.filter(|index| !matches!(index, 
ColumnIndexMetaData::NONE));
+    if column_index.is_some_and(|index| index.num_pages() != pages.len() as 
u64) {
+        return false;
+    }
+    let Some(first) = pages.first() else {
+        return row_count == 0;
+    };
+    let data_start = column.data_page_offset();
+    let chunk_start = column.dictionary_page_offset().unwrap_or(data_start);
+    let Some(chunk_end) = chunk_start.checked_add(column.compressed_size()) 
else {
+        return false;
+    };
+    if !(0..=data_start).contains(&chunk_start)
+        || !u64::try_from(chunk_end).is_ok_and(|end| end <= file_size)
+        || first.first_row_index != 0
+        || first.offset != data_start
+    {
+        return false;
+    }
+    // Arrow reads each indexed page by its recorded size, including the 
header.
+    // A gap can truncate a projected page even when another file supplies the 
filter.
+    pages.iter().enumerate().all(|(index, page)| {
+        let (next_row, next_byte) = pages.get(index + 1).map_or((row_count, 
chunk_end), |next| {
+            (next.first_row_index, next.offset)
+        });
+        page.first_row_index < next_row
+            && page.compressed_page_size > 0
+            && page
+                .offset
+                .checked_add(i64::from(page.compressed_page_size))
+                .is_some_and(|end| end == next_byte)
+            && column_index.is_none_or(|stats| {

Review Comment:
   **[P1] Verify projected page row boundaries before merging a sparse 
selection**
   
   A projected file can have an OffsetIndex without a ColumnIndex or row-count 
histograms. The check accepts exact byte spans and monotone row starts even 
when the recorded row starts disagree with the actual pages. Arrow then uses 
those starts both to fetch pages and to skip records.
   
   I reproduced this with 37-row `id` pages and an OffsetIndex-only file whose 
second page reports row 36 instead of 37. A separate, intact `rating` provider 
with one-row pages prunes `rating = 36` to logical row 100036. With pruning off 
the result is `(id=36, _ROW_ID=100036)`. With pruning on it is `(id=37, 
_ROW_ID=100036)`. The residual sees the correct rating and cannot detect the 
mismatched projection. Please validate the touched pages' row counts or fall 
back to whole-chunk decoding for projected columns whose boundaries cannot be 
independently checked.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to