mbutrovich commented on code in PR #3739:
URL: https://github.com/apache/iceberg-rust/pull/3739#discussion_r4648107743
##########
crates/iceberg/Cargo.toml:
##########
@@ -58,6 +58,7 @@ derive_builder = { workspace = true }
fastnum = { workspace = true }
flate2 = { workspace = true }
fnv = { workspace = true }
+foldhash = { workspace = true }
Review Comment:
> I can revert the last commit if we don't want this new dep.
I'd keep it. DataFusion already depends on [`foldhash =
"0.2"`](https://github.com/apache/datafusion/blob/904afffc731637c4c927c8e095ed07fb195a6c79/datafusion/common/Cargo.toml#L64),
so DataFusion-based consumers already build it, and foldhash has no
dependencies of its own.
Should `Int` and `Long` use `FixedState` too? FNV hashes one byte at a time,
so each `i64` row costs 8 dependent multiplies before the probe starts. This is
the Int64 arm of `probe_set` at a16ffe7, release build on aarch64:
```asm
ldr x0, [x9, x0, lsl #3] ; load the i64
and x1, x0, #0xff
eor x1, x1, x10 ; FNV offset basis
mul x1, x1, x11 ; x11 = FNV prime, round 1 of 8
... ; rounds 2-7
eor x1, x1, x0, lsr #56
mul x2, x1, x11 ; round 8
```
foldhash hashes a single integer with one widening multiply in
[`finish`](https://github.com/orlp/foldhash/blob/8f878c636fda9c9e93384824ea45e06d03f009f5/src/fast.rs#L114-L122).
With `HashSet<i64, FixedState>` the same arm becomes:
```asm
ldr x1, [x9, x1, lsl #3] ; load the i64
eor x2, x1, x10
mul x3, x2, x11
umulh x2, x2, x11 ; independent of the mul above
eor x2, x2, x3
```
Could you add an Int64 case to the reader benchmark and compare the two
hashers? If it holds up, `SetIndex` would also use one hasher for every variant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]