Hi Junjie, I think the vote is ambigous in its current form (can people vote on one option only or can they vote on both?) and has a low chance of getting votes in general because it's not a yes/no question but a choose-an-approach question instead. I think most contributors would accept the hash chosen based on a community discussion but would be reluctant to make that choice themselves in the form a vote because it requires a much deeper dive into the technical intricacies involved. The committers are experienced in the parquet code base but may not be as experienced in bloom filters as you are.
In my opinion, to get bloom filtering into parquet-mr, you should convince the committers that the proposal is viable by addressing their concerns (which I believe you have done), and not by delegating the task of making choices to them. I would suggest that you propose which one (or both) of the hashes should be included, summarize your motivations in this thread and if you don't get any objections for a day or two, call a YES/NO vote for that specific proposal in a separate thread. Thanks, Zoltan On Tue, Jul 2, 2019 at 3:52 AM 俊杰陈 <[email protected]> wrote: > Any thoughts from other committers and developers? > > I 'd like to start a vote firstly, you could either provide your input here > or on vote thread. > > > > On Mon, Jul 1, 2019 at 8:20 PM Zoltan Ivanfi <[email protected]> > wrote: > > > Hi, > > > > I would like to clarify one point of my previous e-mail: While I reasoned > > that for compressions and encodings we should avoid picking algorithms > > superseded by better ones, I also reasoned that for bloom filters we do > not > > necessarily have to be as strict, because a reader with missing > > implementation will still be able to read data from files that contain > > unsupported bloom filter data structures. > > > > Personally I'm fine with moving forward with the current hash proposal, > > even if the chosen algorithm is not considered to be the best of its > class. > > > > Br, > > > > Zoltan > > > > On Sun, Jun 30, 2019 at 11:02 PM Jim Apple <[email protected]> wrote: > > > > > On 2019/06/28 16:43:23, Ryan Blue <[email protected]> wrote: > > > > I agree with Zoltan. Since we want to ensure compatibility, it would > be > > > > better to choose the best option now instead of making everyone > support > > > two > > > > options forever. > > > > > > I'd guess there probably isn't a single best option. I suspect there's > a > > > tradeoff between ease of implementation and speed, for instance, since > I > > > expect it's easy to find an MD5 library in most programming languages > and > > > operating systems, yet MD5 is very slow compared to non-cryptographic > > hash > > > functions designed for speed like xxhash. > > > > > > There's also a significant amount of variability across processor > > families > > > (64-bit multiply-shift in ARM vs x86-64) or even different versions of > > the > > > same processor family (CLHash in Haswell vs. Sandy Lake). There are > also > > > quality tradeoffs that depend on the average bye length of the input > (FNV > > > vs vhash) or how much L1 cache the user wants to use for the hash > > function > > > (tabulation hashing vs. multiply-shift). > > > > > > To deal with this level of ambiguity, I'd suggest that v1 should > include > > a > > > hash function that works well for certain common environments. As far > as > > I > > > know, murmur and xxhash would both fit that bill. > > > > > > > > -- > Thanks & Best Regards >
