SteNicholas commented on code in PR #1469:
URL: https://github.com/apache/paimon-rust/pull/1469#discussion_r4656859830
##########
crates/paimon/src/arrow/format/parquet.rs:
##########
@@ -88,9 +94,90 @@ impl ParquetFormatReader {
page_index_enabled:
core_options.parquet_filter_column_index_enabled()?,
// Format-table files may be replaced in place.
metadata_cache_enabled: !core_options.is_format_table(),
+ data_evolution_enabled: core_options.data_evolution_enabled(),
+ metadata: None,
})
}
+ pub(crate) fn with_metadata(mut self, metadata: ArrowReaderMetadata) ->
Self {
+ self.metadata = Some(metadata);
+ self
+ }
+
+ fn reader_options(&self, has_predicate: bool, has_selection: bool) ->
ArrowReaderOptions {
+ let mut options = ArrowReaderOptions::new();
+ if self.page_index_enabled {
+ if has_predicate || (self.data_evolution_enabled && has_selection)
{
Review Comment:
Fixed in
[b6d10803](https://github.com/apache/paimon-rust/commit/b6d10803a153c667a92434148659e054607f95a9).
Predicate-free sparse reads load only OffsetIndex, with merged-read pruning
either enabled or disabled. If loading optional column and offset indexes
together fails, the reader retries offsets independently before falling back to
the footer. The retry reuses already-fetched metadata bytes; required
ColumnIndex errors still propagate.
A counting-I/O regression uses 30,000 one-row pages so the indexes extend
beyond footer prefetch. Valid and corrupt optional ColumnIndexes now produce
the same row, metadata bytes and column-data bytes as the ordinary
OffsetIndex-only path. A separate regression verifies that optional ColumnIndex
failure preserves the valid offsets with one metadata read. [I/O
regression](https://github.com/apache/paimon-rust/blob/b6d10803a153c667a92434148659e054607f95a9/crates/paimon/src/arrow/format/parquet.rs#L6975)
ยท [Retry
regression](https://github.com/apache/paimon-rust/blob/b6d10803a153c667a92434148659e054607f95a9/crates/paimon/src/arrow/format/parquet.rs#L7039)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]