alamb opened a new pull request, #8294: URL: https://github.com/apache/arrow-datafusion/pull/8294
## Which issue does this PR close? Part of https://github.com/apache/arrow-datafusion/issues/8229 In order to avoid boiling the ocean and to document more clearly what the current code does, this PR: ## Rationale for this change I am in the process of trying to improve the statistics in DataFusion, which have grown organically over time. I would like to refactor them, but I need to ensure that I don't break anything. There are tests for the existing pruning predicate code, but not the underlying statistics conversion. There are a few problems with the existing code: 1. There are at least two copies of code that converts parquet statistics into DataFusion statistics, which have somewhat different semantics (TODO links) 2. Currentky the statistics areconverted one row at a time (as an Array), which is not ideal from evaluating the pruning statistics <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? 1. Extracts the statistics conversion code to a new module, and add a columnar API (returns value as an ArrayRef). 2. Adds extensive tests, both round tripping data through `parquet` rust writer as well as using the existing parquet test data <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? Yes <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? There are non intended. This implementation uses the same the existing code, so it is not a functional change, but it does add many tests for the existing code. I plan to improve the existing code in follow on PRs. <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
