unikdahal commented on issue #3343: URL: https://github.com/apache/iceberg-rust/issues/3343#issuecomment-6129268588
Hi all, a quick heads-up: I'll be away for about a month starting now, and I don't want this epic to block anyone in the meantime. If any part of it is useful to you, please feel free to pick it up, rework it, or take it in a different direction. There's no need to wait for me or ask first. Here's where things stand, to make that easier: - **#3345** (lazy delete-file sizes): @JeonDaehong's review is right. Since #3145 already changes `FileScanTaskDeleteFile` for 0.11, modelling an unknown size as `Option<u64>` is cleaner than the `0` sentinel. The PR needs a rebase on #3145 plus that change. It's small, if someone wants an easy start. - **#3347** (inclusive metrics evaluator over borrowed statistics): small and self-contained, but nothing calls it yet. Its first caller would be the planned follow-up that rejects file tasks from retained statistics, so it may be easier to review together with that. - **#3351** (execution-time predicate API + task-start pruning): still waiting for review. It only works through `ArrowReaderBuilder` today, not `TableScan::to_arrow`, so wiring it into `TableScan` would be a natural follow-up. - **Live refresh at row-group boundaries** depends on the push-decoder APIs in Arrow 60 (#3257), so that part has to wait for the upgrade either way. A few things I learned from an end-to-end prototype (iceberg-rust plus DataFusion Comet), in case they save someone time: - **Keep runtime predicates advisory.** A bind, planning or evaluation failure should only drop the runtime restriction, never the planned filter or delete handling. - **Bind once per generation, not once per task.** Binding a large `IN` list for every file adds up quickly. A monotonic generation from the publisher keeps the cache simple and safe. - **Floats need care.** Iceberg statistics and Arrow kernels disagree on NaN and signed zero, so I declined float runtime predicates and guarded the "all rows match" shortcut for floats. - **Large membership sets** benefit from hash lookups in the row filter (binary and decimal included), and a task needs a cap on how much replanning it does when predicates change often. - **Position deletes and `_pos` / `_row_id`** have to keep original file ordinals through all of the pruning. I also just saw apache/datafusion-iceberg#68, which proposes a very similar hook. If that design, or its author, ends up driving this, I'm completely happy for #3351 to be closed in favour of it. What matters is that the feature lands, not whose PR it is. Thanks to everyone who has weighed in on the related issues and PRs, and sorry for leaving this half-finished for a few weeks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
