coady opened a new issue, #36127:
URL: https://github.com/apache/arrow/issues/36127
### Describe the enhancement requested
`Dataset.get_fragments` on a filtered dataset has differing behavior:
* `dataset.get_fragments(filter=expr)` works on partition keys, and appears
to ignore any portion of the expression that refers to other fields (which can
have surprising results)
* `dataset.filter(expr).get_fragments()` always raises an error
```python
# Accessing fragments of a filtered dataset is not supported.
# It would be unclear if you wanted to filter the fragments
# or the rows in those fragments.
raise ValueError(
"Retrieving fragments of a filtered or projected "
"dataset is not allowed. Remove the filtering."
)
```
But it's not really relevant whether the filter is previously applied or
passed in the call. The question is what is the right behavior if the
expression refers to non-partition keys. A compromise proposal: allow filtering
on only partition keys, otherwise raise an error.
And if that behavior is acceptable, then the `filter` parameter in
`get_fragments` (which pre-dates general filtering) looks redundant, and could
be deprecated.
### Component(s)
C++, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]