brgr-s opened a new issue, #3370: URL: https://github.com/apache/iceberg-rust/issues/3370
### Is your feature request related to a problem or challenge? This is a follow-up for #2962, split out to keep that PR small. See https://github.com/apache/iceberg-rust/pull/2961#discussion_r3722147076. After #2961, equality deletes are applied as a hash lookup inside the parquet `RowFilter`. The probe still does more work per row than it needs to. For every batch and every key column it: - builds a `Vec<Option<Literal>>` for the whole column via `arrow_primitive_to_literal`, - wraps each cell in a `Datum` (and promotes it with `Datum::to` when the file type differs from the table type), - hashes the resulting `EqDeleteKey(Vec<Option<Datum>>)` with the default SipHash of `std::collections::HashSet`. `arrow::row::RowConverter` (crate `arrow-row`, already in `Cargo.lock`, not yet a direct dependency of `crates/iceberg`) builds comparable, hashable rows straight from Arrow arrays without per-cell boxing. DataFusion uses it for multi-column join and group keys. For the common single-column case, DataFusion's `InListExpr` static filters use a native-typed `hashbrown`/`ahash` set instead. The delete side (`caching_delete_file_loader.rs`, parsing delete files into `EqDeleteSet`) has the same pattern, but runs once per delete file, so it matters less. ### Describe the solution you'd like - Benchmark the current probe as the baseline (single `long` key; `long` + short string key; with and without type promotion). - Store delete keys as `RowConverter` rows (or a native-typed set for one column) and probe with rows built from the batch. - Keep current semantics: null matches null, type promotion under schema evolution, and float canonicalisation (`-0.0 == 0.0`, all NaNs equal). - Land only if the benchmark shows a clear win. ### Willingness to contribute I cannot contribute to this feature at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
