xudong963 commented on PR #21820: URL: https://github.com/apache/datafusion/pull/21820#issuecomment-4987861192
@zhuqi-lucas Thanks! Current HEAD already has custom-hasher coverage for Dictionary and checks equivalence with the underlying Utf8 values. I’ll add coverage for Struct and long Utf8View values, including the external-buffer path. For parity, I can add scoped single-column leaf tests using the same RandomState. Full bit-for-bit parity is not currently a valid invariant: the optimized HashState path seeds primitive/view rehashing from the previous column hash, while the generic BuildHasher path uses combine_hashes. Multi-column inputs and multi-field structs can therefore differ even with the same builder. I’ll document that distinction; if full parity is expected, we should align on the API design first. Both null helpers currently use mul_col; I’ll rename both to multi_col. The near-term consumer is from Massive's latest stateful streaming system that hashes Arrow group-key columns with a fixed-seed XXH3 builder for stable key-group and checkpoint routing. (The project is still private). I guess if @AdamGS has some other use cases. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
