sunchao opened a new pull request, #5453: URL: https://github.com/apache/datafusion-comet/pull/5453
## Why are the changes needed? Native Parquet scans currently expose aggregate scan metrics, but those metrics do not show how much I/O came from projected data pages versus footer, page-index, or Bloom-filter metadata. They also do not distinguish bytes requested from an object store from response bytes actually consumed, or explain whether repeated metadata access was served from cache. This makes it difficult to diagnose read amplification, excessive metadata traffic, and the effect of projection, predicate pruning, and metadata caching. For example, a scan might return 100 MiB of projected data while consuming 150 MiB from object storage. Existing aggregate scan metrics cannot show whether the additional 50 MiB came from metadata, coalesced range requests, or other over-read. Separate reader-level and object-store-level counters make that 1.5x observed read amplification visible without conflating local/cache reads with network I/O. ## What changes were proposed in this PR? Add nine native Parquet scan metrics and expose them as Spark SQL metrics: - `scan_io_data_bytes`: projected Parquet data-page bytes returned to the reader. - `scan_io_metadata_bytes`: footer-prefetch, page-index, and Bloom-filter bytes returned to the reader. - `scan_io_footer_reads`: serialized Parquet footer payloads read from storage. - `scan_io_footer_bytes`: serialized Parquet footer payload bytes read from storage. - `scan_io_object_store_get_calls`: object-store GET operations after range coalescing. - `scan_io_object_store_get_requested_bytes`: object-store GET range bytes requested after range coalescing. - `scan_io_object_store_response_bytes_read`: object-store response bytes actually consumed after range coalescing. - `scan_io_metadata_cache_hits`: successful metadata loads served without storage I/O. - `scan_io_metadata_cache_misses`: successful metadata loads requiring storage I/O. The reader-level counters classify data and metadata independently, while the object-store counters are recorded only for recognized remote object stores; local files and unsupported/ambiguous stores are not mislabeled as network traffic. Footer bytes are a subset of metadata bytes, not an additional category to sum. Metadata cache outcomes are inferred from observed storage reads rather than assumed from cache configuration. Instrumentation wraps coalesced object-store requests and counts response chunks as they are consumed, avoiding double counting logical ranges and handling partial consumption. Footer accounting is idempotent and remains accurate for cached metadata, encrypted files, invalid metadata, and page-index failures. The change also keeps native producer shutdown bounded and isolates object-store registrations so metric publication and source classification remain correct during early termination and mixed-store execution. ## How was this PR tested? - Added native fixture-based coverage for cold and warm metadata caching, metadata-only reads, projected data reads, page-index and Bloom-filter metadata, coalesced object-store ranges, local versus remote storage, encrypted footers, malformed footers, page-index failures, early termination, and object-store registration isolation. - Added Spark integration assertions that all nine metrics are registered, reader-level counters are populated, and object-store counters remain zero for local scans. - Re-ran `cargo fmt --all -- --check` and `cargo test -p datafusion-comet --lib parquet::parquet_exec::tests` on the published head; all 16 focused native tests passed. - Ran native Rust library tests with default features and without default features, plus default/no-default Clippy with warnings denied and `cargo fmt --all -- --check`. - Ran focused Spark native-scan and collect-limit coverage with Spark 3.4, 3.5, 4.0, 4.1, and 4.2; ran `CometTaskMetricsSuite` with Spark 3.5, 4.0, and 4.2; and ran ScalaStyle and Spotless checks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
