JeonDaehong commented on PR #18027: URL: https://github.com/apache/iceberg/pull/18027#issuecomment-5636284642
@pvary I’ve resent the email as well! Here is a quick summary for you here too :D **TL;DR** — yes, 13-19% is accurate and if anything conservative. But the summary dropped three conditions, and one of them is backwards: more deletes is *not* more benefit. I've rewritten the PR description to lead with when this helps. Thanks for both points. ### Is 13-19% accurate? It's measured rather than extrapolated, on projections of 5 columns or fewer, all integer. A scan-only query with a single projected column came out at 29% on a dedicated instance. At 10 columns or more it falls inside my noise floor (9.7% median run-to-run spread) and I don't claim anything there. ### When it helps | | | |---|---| | **Helps** | Narrow projection **and** the columns are integers | | | Scan-dominated query, little aggregation | | | Delete rate near the array/bitmap boundary (~6.25%) | | | Fewer, larger files | | **Doesn't help** | One md5 string column costs 8-10 integer columns of decode, which sinks it on its own | | | Heavy aggregation — the CPU saved is constant, so its share falls to ~3% as the aggregation grows | | | Many small files — per-file DV load cost dilutes it (60.7% of scan CPU at 4 files, 49.7% at 488) | ### The one that's backwards "DV-heavy" reads as "more deletes, more benefit", and it's the other way round. The worst case — and so the largest gain — is just *below* 6.25%, where the Roaring container is still a sorted array at its deepest. Past that it flips to a bitmap and gets cheaper on its own. Deleting more rows can make the scan faster, which is the least intuitive thing I found here. ### One rule instead of six The patch is worth it when the delete check is more than **~21% of scan CPU**. That single threshold predicted the crossover better than column count or column type did. --- Point taken on length — I've added a "when this helps" paragraph at the top of the description here, and I'll lead with a TL;DR on the list. Sorry it landed in your spam folder; there's a thread on the dev list about devlist mail going to gmail spam, so it may not be specific to my message. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
