mixermt commented on PR #7712:
URL: 
https://github.com/apache/datafusion-comet/pull/7712#issuecomment-7063716754

   Thanks for the careful review. I've pushed changes for each point.
   
   **Cost**
   
   - Setting `spark.comet.scan.icebergNative.narrowTypeCheck.enabled` to 
`false` turns the check off. It defaults to `true`.
   - The reader now reuses the footer the check read. The scan gives 
iceberg-rust a `FileIO` that serves those bytes from memory until each split of 
the file has opened it, so the reader doesn't fetch the footer a second time. 
Both sides use the same 512 KiB metadata size hint, so the reader asks for the 
same footer bytes the check already has.
   
   What's left is one extra footer decode per checked file. On MinIO with 20 ms 
added to each request (median of 5 runs, check off → on):
   
   | Table | Query | GET requests | Time |
   |---|---|---|---|
   | 400 files of ~60 KB | `a < 0.001` | 729 → 365 | 2475 → 1300 ms |
   | 16 files of ~2.4 MB | `a < 0.001` | 33 → 33 | 264 → 251 ms |
   
   Small files come out faster than today, because the check's read covers the 
whole file and the reader serves its data reads from the same bytes. Before 
footer sharing, the same small-file query was 27% slower with the check on.
   
   **1. `Not`**
   
   Comet can't send one: `planner.rs` calls `rewrite_not()` before binding. 
Binding itself keeps `Not`, though, and the classification test binds `!(id = 
1)` without rewriting, so that arm did run, but only for `Eq`. It now negates 
each comparison the way `rewrite_not` does, so `NOT (d IS NULL)` counts as safe 
instead of at risk. I added `In`, `IsNull`, and double `NOT` cases.
   
   **2. Page index**
   
   It was covered, though not obviously: the page index skips a file's only 
page too. With the check disabled, the first query to fail in each existing 
test is `id > 0` or `id > 1`. Those literals fit in `INT`, so the literal cast 
is fine and only page skipping can drop the rows. With row selection also 
disabled, those queries return the right rows. I added a test whose file stores 
`id` in 6 pages, with in-range literals only. With the check off, it returns 0 
of 499 rows.
   
   **#7156 and #7233**
   
   This replaces #7233 and the promoted-column rule in #7156. Because it checks 
each file rather than the whole table, files that store the table's type keep 
their pruning. It also catches files the metadata doesn't record, such as 
promotions made before Iceberg 0.12 and files appended without a name mapping. 
The rest of #7156, the float/NaN and `NOT` fixes for #7136 and #7137, is 
independent and still needed. Its `timestamp` rule is preventive, since Iceberg 
1.11 doesn't allow promoting `date` yet. If #7156 lands first, I can remove its 
promotion rule in this PR, or #7156 can drop it. Whichever you prefer.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to