sunchao commented on code in PR #7598: URL: https://github.com/apache/datafusion-comet/pull/7598#discussion_r4575416813
########## native/core/src/execution/operators/dynamic_filter/min_max/row_group_limit.rs: ########## @@ -0,0 +1,119 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! Limit the work performed by DataFusion's live row-group pruner. + +use std::sync::Arc; + +use datafusion::datasource::physical_plan::FileSource; +use datafusion::datasource::source::DataSourceExec; +use datafusion::physical_plan::ExecutionPlan; + +use super::super::parquet_reader::concrete_parquet_source; +use crate::parquet::file_error_context::ParquetErrorContext; +use crate::parquet::parquet_exec::METADATA_SIZE_HINT; + +mod reader; +use reader::{PreflightFileIndex, PreloadedReaderFactory}; + +// DataFusion 55.2 rechecks the entire unread tail at every row-group boundary, +// even when the predicate has not changed. Until that work is bounded upstream, +// cap each eligible file rather than merely throttling aggregate updates. +pub(super) const MAX_ROW_GROUPS_PER_FILE: usize = 64; + +// Retaining metadata must not turn the preflight into an unbounded cache. Stop +// after crossing this budget, retaining that last footer for the ordinary scan. +pub(super) const MAX_PREFLIGHT_METADATA_BYTES: usize = 8 * 1024 * 1024; + +pub(super) async fn prepare_scan( + input: &Arc<dyn ExecutionPlan>, + filtered_input: &Arc<dyn ExecutionPlan>, + partition: usize, +) -> (Arc<dyn ExecutionPlan>, bool) { + let Some(scan) = input.downcast_ref::<DataSourceExec>() else { + return (Arc::clone(input), false); + }; + let Some(original) = concrete_parquet_source(scan) else { + return (Arc::clone(input), false); + }; + let Some(filtered_scan) = filtered_input.downcast_ref::<DataSourceExec>() else { + return (Arc::clone(input), false); + }; + let Some(filtered) = concrete_parquet_source(filtered_scan) else { + return (Arc::clone(input), false); + }; + let (config, source, _) = original; + let Some(factory) = source.parquet_file_reader_factory() else { + return (Arc::clone(input), false); + }; + let Some(group) = config.file_groups.get(partition) else { + return (Arc::clone(input), false); + }; + let mut readers = PreloadedReaderFactory::new(Arc::clone(factory)); + let mut metadata_bytes = 0; + let mut within_limit = true; + // Hand successful reads to either scan, including when a later file makes + // us fall back. Let normal decoding report metadata errors in its original + // order and context. Keeping each reader also preserves its validation state + // and avoids starting its next-file prefetch a second time. + for file in group.files() { + let Ok(mut reader) = factory.create_reader( + partition, + file.clone(), + file.metadata_size_hint.or(Some(METADATA_SIZE_HINT)), + source.metrics(), + ) else { + within_limit = false; + break; + }; + let Ok(metadata) = reader.get_metadata(None).await else { Review Comment: [P2] Skip MIN/MAX preflight when the Parquet scan uses decryption Preflight calls `get_metadata(None)`, so it reads the encrypted footer before discovering it has no decryption properties. The fallback scan then opens the same file with the key and reads the footer again. The successful-reader handoff only retains readers whose preflight metadata load succeeded, so it cannot cover this case. The encrypted scan already disables its own metadata prefetch for the same reason. I reproduced this on `c91a120` with one file, one row group, an encrypted footer and a fixed test key. Ordinary partial MIN and the native wrapper both returned `1`. With an 8 KiB footer entry, the instrumented reader counted 9,383 metadata bytes on the ordinary scan and 18,766 with the wrapper. With an 8 MiB entry it counted 8,913,539 and 17,827,078 bytes, respectively. The extra read produces no pruning. Carry the scan’s encryption eligibility into the wrapper so encrypted scans can bypass preflight before opening the file. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
