mbutrovich commented on code in PR #3739:
URL: https://github.com/apache/iceberg-rust/pull/3739#discussion_r4648107743


##########
crates/iceberg/Cargo.toml:
##########
@@ -58,6 +58,7 @@ derive_builder = { workspace = true }
 fastnum = { workspace = true }
 flate2 = { workspace = true }
 fnv = { workspace = true }
+foldhash = { workspace = true }

Review Comment:
   > I can revert the last commit if we don't want this new dep.
   
   I'd keep it. DataFusion already depends on [`foldhash = 
"0.2"`](https://github.com/apache/datafusion/blob/904afffc731637c4c927c8e095ed07fb195a6c79/datafusion/common/Cargo.toml#L64),
 so DataFusion-based consumers already build it, and foldhash has no 
dependencies of its own.
   
   Should `Int` and `Long` use `FixedState` too? FNV hashes one byte at a time, 
so each `i64` row costs 8 dependent multiplies before the probe starts. This is 
the Int64 arm of `probe_set` at a16ffe7, release build on aarch64:
   
   ```asm
   ldr  x0, [x9, x0, lsl #3]   ; load the i64
   and  x1, x0, #0xff
   eor  x1, x1, x10            ; FNV offset basis
   mul  x1, x1, x11            ; x11 = FNV prime, round 1 of 8
   ...                         ; rounds 2-7
   eor  x1, x1, x0, lsr #56
   mul  x2, x1, x11            ; round 8
   ```
   
   foldhash hashes a single integer with one widening multiply in 
[`finish`](https://github.com/orlp/foldhash/blob/8f878c636fda9c9e93384824ea45e06d03f009f5/src/fast.rs#L114-L122).
 With `HashSet<i64, FixedState>` the same arm becomes:
   
   ```asm
   ldr   x1, [x9, x1, lsl #3]  ; load the i64
   eor   x2, x1, x10
   mul   x3, x2, x11
   umulh x2, x2, x11           ; independent of the mul above
   eor   x2, x2, x3
   ```
   
   Could you add an Int64 case to the reader benchmark and compare the two 
hashers? If it holds up, `SetIndex` would also use one hasher for every variant.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to