Hi, I would like to clarify one point of my previous e-mail: While I reasoned that for compressions and encodings we should avoid picking algorithms superseded by better ones, I also reasoned that for bloom filters we do not necessarily have to be as strict, because a reader with missing implementation will still be able to read data from files that contain unsupported bloom filter data structures.
Personally I'm fine with moving forward with the current hash proposal, even if the chosen algorithm is not considered to be the best of its class. Br, Zoltan On Sun, Jun 30, 2019 at 11:02 PM Jim Apple <[email protected]> wrote: > On 2019/06/28 16:43:23, Ryan Blue <[email protected]> wrote: > > I agree with Zoltan. Since we want to ensure compatibility, it would be > > better to choose the best option now instead of making everyone support > two > > options forever. > > I'd guess there probably isn't a single best option. I suspect there's a > tradeoff between ease of implementation and speed, for instance, since I > expect it's easy to find an MD5 library in most programming languages and > operating systems, yet MD5 is very slow compared to non-cryptographic hash > functions designed for speed like xxhash. > > There's also a significant amount of variability across processor families > (64-bit multiply-shift in ARM vs x86-64) or even different versions of the > same processor family (CLHash in Haswell vs. Sandy Lake). There are also > quality tradeoffs that depend on the average bye length of the input (FNV > vs vhash) or how much L1 cache the user wants to use for the hash function > (tabulation hashing vs. multiply-shift). > > To deal with this level of ambiguity, I'd suggest that v1 should include a > hash function that works well for certain common environments. As far as I > know, murmur and xxhash would both fit that bill. >
