adriangb commented on PR #9628: URL: https://github.com/apache/arrow-rs/pull/9628#issuecomment-4162471419
> > > Yea, I think we can deprecate the old behavior and maybe remove it after few releases. > > > > > > Do you want to do that in this PR or in a followup (maybe once this is out in the wild and known to be working well)? > > I think we can do it in this PR. If we want to deprecate the existing NDV I think we're better off re-interpreting it to mean "maximum ndv" or "initial ndv". That way existing users who are setting the ndv also benefit from folding. This means there will be no way to disable folding but I also don't see any reason anyone would want to do that beyond requiring a fixed-size bloom filter (in which case relying on a combination of fpp + ndv giving you a fixed size was probably a bad choice to begin with given I don't think we made any such API promise, and they should open an issue requesting an explicit API for this). Thus the only changes vs. main now are: 1. Adding the folding on write. 2. Default max ndv is derived from the max rows per row group instead of being hardcoded. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
