dwsmith1983 opened a new issue, #6039: URL: https://github.com/apache/datafusion-comet/issues/6039
### What is the problem the feature request solves? The native Iceberg scan skips Parquet pages: `IcebergScanExec` enables iceberg-rust's row selection and passes the residual predicate, so iceberg-rust loads the page index and builds a page-level row selection. Measured locally on one data file with one row group and 158 `id` pages, `WHERE id BETWEEN 1000 AND 1100` reads about 550 KB of a 21 MB file. Nothing in the test suite would notice if that stopped. The behaviour hinges on one builder flag in the scan operator and on iceberg-rust internals that change with every pin bump, and `CometIcebergNativeScanExec` reports no page-index counters (iceberg-rust's scan metrics expose only bytes read), so a regression would show up only as slower queries. ### Describe the potential solution A regression test in `CometIcebergNativeSuite` that writes a single-row-group, many-page table, verifies that layout from the footer so the fixture cannot degrade into a shape where file-level or row-group pruning explains the savings, and asserts that a narrow range reads a small fraction of the full read's `bytes_scanned`. ### Additional context Page-index counters on the Iceberg node, matching the Parquet scan's `page_index_pages_pruned`, need iceberg-rust to expose them first. Related: #6026. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
