mixermt commented on PR #7712: URL: https://github.com/apache/datafusion-comet/pull/7712#issuecomment-7063716754
Thanks for the careful review. I've pushed changes for each point. **Cost** - Setting `spark.comet.scan.icebergNative.narrowTypeCheck.enabled` to `false` turns the check off. It defaults to `true`. - The reader now reuses the footer the check read. The scan gives iceberg-rust a `FileIO` that serves those bytes from memory until each split of the file has opened it, so the reader doesn't fetch the footer a second time. Both sides use the same 512 KiB metadata size hint, so the reader asks for the same footer bytes the check already has. What's left is one extra footer decode per checked file. On MinIO with 20 ms added to each request (median of 5 runs, check off → on): | Table | Query | GET requests | Time | |---|---|---|---| | 400 files of ~60 KB | `a < 0.001` | 729 → 365 | 2475 → 1300 ms | | 16 files of ~2.4 MB | `a < 0.001` | 33 → 33 | 264 → 251 ms | Small files come out faster than today, because the check's read covers the whole file and the reader serves its data reads from the same bytes. Before footer sharing, the same small-file query was 27% slower with the check on. **1. `Not`** Comet can't send one: `planner.rs` calls `rewrite_not()` before binding. Binding itself keeps `Not`, though, and the classification test binds `!(id = 1)` without rewriting, so that arm did run, but only for `Eq`. It now negates each comparison the way `rewrite_not` does, so `NOT (d IS NULL)` counts as safe instead of at risk. I added `In`, `IsNull`, and double `NOT` cases. **2. Page index** It was covered, though not obviously: the page index skips a file's only page too. With the check disabled, the first query to fail in each existing test is `id > 0` or `id > 1`. Those literals fit in `INT`, so the literal cast is fine and only page skipping can drop the rows. With row selection also disabled, those queries return the right rows. I added a test whose file stores `id` in 6 pages, with in-range literals only. With the check off, it returns 0 of 499 rows. **#7156 and #7233** This replaces #7233 and the promoted-column rule in #7156. Because it checks each file rather than the whole table, files that store the table's type keep their pruning. It also catches files the metadata doesn't record, such as promotions made before Iceberg 0.12 and files appended without a name mapping. The rest of #7156, the float/NaN and `NOT` fixes for #7136 and #7137, is independent and still needed. Its `timestamp` rule is preventive, since Iceberg 1.11 doesn't allow promoting `date` yet. If #7156 lands first, I can remove its promotion rule in this PR, or #7156 can drop it. Whichever you prefer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
