Thank you all for making the next step clear. I also agree to choose the best option.
The changes to parquet.thrift and BloomFilter.md is ready now in this PR <https://github.com/apache/parquet-format/pull/139>. Please take a look and then we can move forward to VOTE. On Sat, Jun 29, 2019 at 12:44 AM Ryan Blue <[email protected]> wrote: > I agree with Zoltan. Since we want to ensure compatibility, it would be > better to choose the best option now instead of making everyone support two > options forever. > > In terms of next steps, I think that getting a clean write-up of the design > and changes and starting a VOTE thread that points to them are the next > steps. The write-up is already done, but needs to be updated for xxHash, > right? > > rb > > On Fri, Jun 28, 2019 at 3:59 AM Zoltan Ivanfi <[email protected]> > wrote: > > > Hi, > > > > I think the concern was not about the lack of any specific hash > algorithm, > > but about the choice of the one that got added. Generally for > compressions > > and encodings, we are very picky about which ones to add to > specification, > > because it has to be implemented in every language binding. This is not > > only a considerable effort, but is also error-prone (see LZ4 for an > > example, which was added to both the Java and the C++ implementation of > > Parquet, yet they are incompatible with each other). And lack of support > is > > not only a minor annoyance in this case: if one is forced to use an older > > reader that does not support the new encoding yet (or a language binding > > that does not support it at all), the data simply can not be read. > > > > For this reason, if we already know that an algorithm is suboptimal and > > there are better ones available, we prefer not to add it at all. > However, I > > don't think that the reasoning above applies here though, because the > bloom > > filter is an optional metadata and the data is perfectly readable without > > supporting it. Even if it is very likely that we will want to move to a > > better hash algorithm later, we already know that we won't have to keep > > supporting the current one forever, since removing support is not a > > breaking change (at least functionally, performance-wise it will result > in > > a regression for old files). > > > > Br, > > > > Zoltan > > > > On Fri, Jun 28, 2019 at 11:12 AM 俊杰陈 <[email protected]> wrote: > > > > > Thanks, > > > > > > The naming issue had been fixed, I also created a PR > > > <https://github.com/apache/parquet-format/pull/139>to add xxHash as an > > > alternative option for Todd's concern. Is that OK for concerns? If that > > is > > > OK, we can create a VOTE against the spec (the bloom filter diff in > > > parquet-format repo). > > > > > > On Fri, Jun 28, 2019 at 4:03 PM Driesprong, Fokko <[email protected] > > > > > wrote: > > > > > > > Ryan has a valid point here. Once the Bloom filters get released, it > > > won't > > > > be as easy anymore to change it because we will break an already > > released > > > > API. > > > > > > > > There was a related discussion a while ago: > > > > > > > > > > > > > > https://lists.apache.org/thread.html/027e9d73093df84448e07d8514b9d669906cd5b83ae59a76f38aaa55@%3Cdev.parquet.apache.org%3E > > > > > > > > My suggestion would be to create a VOTE to formally adopt the vote > and > > > fix > > > > the remaining concerns. For example, the one that Zoltan raised in > the > > > list > > > > above. > > > > > > > > Cheers, Fokko > > > > > > > > Op vr 28 jun. 2019 om 01:13 schreef Jim Apple <[email protected]>: > > > > > > > > > > I think we need to have a vote on the bloom filter > > > > > > structures first. We need to make sure that the community has > > vetted > > > > the > > > > > > design and is comfortable with adding this, just like we did with > > the > > > > > > Parquet encryption design and the page index design. > > > > > > > > > > Thank you for the note, Ryan. Based on my experience on Apache > > Impala, > > > I > > > > > was under the impression that a git commit signified at least a > > > temporary > > > > > agreement that the commit should make it into a future release. I > > > > > understand you to be saying that in parquet-format, a vote on > format > > > > > additions is standard, whether or not a commit made it into HEAD. > > > > > > > > > > There have been previous discussions of Bloom filters in the pull > > > > > requests, on this list, and in live videochat meetups (from quite a > > > while > > > > > ago). In your opinion, should we start a new discussion, or start a > > > > [VOTE] > > > > > thread with pointers to the old discussions, or some third option? > > > > > > > > > > > > > > > > > > -- > > > Thanks & Best Regards > > > > > > > > -- > Ryan Blue > Software Engineer > Netflix > -- Thanks & Best Regards
