adamreeve opened a new pull request, #55665: URL: https://github.com/apache/arrow/pull/55665
### Rationale for this change `Dataset.filter()` stores a filter on the Python Dataset wrapper, which is later used to build scan options. But a `UnionDataset` is represented in C++, so filtered Datasets could not be unioned. See https://github.com/apache/arrow/issues/55447 for more context. ### What changes are included in this PR? This adds a new `FilteredDataset` class that holds a filter in the C++ layer, and converts any datasets with filters to a `FilteredDataset` when they are used to create a `UnionDataset`. The Dataset's filter is combined with any filter in the scan options for predicate pushdown. Scanning a `FilteredDataset` generates `FilteredFragment` instances, which apply the filter when generating batches. The row-level filter doesn't account for any additional filter in the scan options, because that's always applied after the scan, so this would just duplicate work. A possible future optimization could be to implement what was proposed in #28944 so a combined filter could be applied once within the fragment, and then a guarantee declared that uses the combined filter expression, so the later filter could be simplified. ### Are these changes tested? Yes, this includes new C++and Python unit tests. ### Are there any user-facing changes? This adds new types to the public API and new functionality. You can now create a `UnionDataset` using Datasets that have filters, which would previously raise an error. ### Was AI used for this PR? In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR. **PR code and description written by:** - [x] Human - [x] AI (I used Claude Opus 5.5 for the initial implementation, then reviewed and edited everything myself) **Reviewed before submission by:** - [x] Human - [x] AI - [ ] Not reviewed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
