I agree with Zoltan. Since we want to ensure compatibility, it would be better to choose the best option now instead of making everyone support two options forever.
In terms of next steps, I think that getting a clean write-up of the design and changes and starting a VOTE thread that points to them are the next steps. The write-up is already done, but needs to be updated for xxHash, right? rb On Fri, Jun 28, 2019 at 3:59 AM Zoltan Ivanfi <[email protected]> wrote: > Hi, > > I think the concern was not about the lack of any specific hash algorithm, > but about the choice of the one that got added. Generally for compressions > and encodings, we are very picky about which ones to add to specification, > because it has to be implemented in every language binding. This is not > only a considerable effort, but is also error-prone (see LZ4 for an > example, which was added to both the Java and the C++ implementation of > Parquet, yet they are incompatible with each other). And lack of support is > not only a minor annoyance in this case: if one is forced to use an older > reader that does not support the new encoding yet (or a language binding > that does not support it at all), the data simply can not be read. > > For this reason, if we already know that an algorithm is suboptimal and > there are better ones available, we prefer not to add it at all. However, I > don't think that the reasoning above applies here though, because the bloom > filter is an optional metadata and the data is perfectly readable without > supporting it. Even if it is very likely that we will want to move to a > better hash algorithm later, we already know that we won't have to keep > supporting the current one forever, since removing support is not a > breaking change (at least functionally, performance-wise it will result in > a regression for old files). > > Br, > > Zoltan > > On Fri, Jun 28, 2019 at 11:12 AM 俊杰陈 <[email protected]> wrote: > > > Thanks, > > > > The naming issue had been fixed, I also created a PR > > <https://github.com/apache/parquet-format/pull/139>to add xxHash as an > > alternative option for Todd's concern. Is that OK for concerns? If that > is > > OK, we can create a VOTE against the spec (the bloom filter diff in > > parquet-format repo). > > > > On Fri, Jun 28, 2019 at 4:03 PM Driesprong, Fokko <[email protected]> > > wrote: > > > > > Ryan has a valid point here. Once the Bloom filters get released, it > > won't > > > be as easy anymore to change it because we will break an already > released > > > API. > > > > > > There was a related discussion a while ago: > > > > > > > > > https://lists.apache.org/thread.html/027e9d73093df84448e07d8514b9d669906cd5b83ae59a76f38aaa55@%3Cdev.parquet.apache.org%3E > > > > > > My suggestion would be to create a VOTE to formally adopt the vote and > > fix > > > the remaining concerns. For example, the one that Zoltan raised in the > > list > > > above. > > > > > > Cheers, Fokko > > > > > > Op vr 28 jun. 2019 om 01:13 schreef Jim Apple <[email protected]>: > > > > > > > > I think we need to have a vote on the bloom filter > > > > > structures first. We need to make sure that the community has > vetted > > > the > > > > > design and is comfortable with adding this, just like we did with > the > > > > > Parquet encryption design and the page index design. > > > > > > > > Thank you for the note, Ryan. Based on my experience on Apache > Impala, > > I > > > > was under the impression that a git commit signified at least a > > temporary > > > > agreement that the commit should make it into a future release. I > > > > understand you to be saying that in parquet-format, a vote on format > > > > additions is standard, whether or not a commit made it into HEAD. > > > > > > > > There have been previous discussions of Bloom filters in the pull > > > > requests, on this list, and in live videochat meetups (from quite a > > while > > > > ago). In your opinion, should we start a new discussion, or start a > > > [VOTE] > > > > thread with pointers to the old discussions, or some third option? > > > > > > > > > > > > > -- > > Thanks & Best Regards > > > -- Ryan Blue Software Engineer Netflix
