coady opened a new issue, #36127:
URL: https://github.com/apache/arrow/issues/36127

   ### Describe the enhancement requested
   
   `Dataset.get_fragments` on a filtered dataset has differing behavior:
   * `dataset.get_fragments(filter=expr)` works on partition keys, and appears 
to ignore any portion of the expression that refers to other fields (which can 
have surprising results)
   * `dataset.filter(expr).get_fragments()` always raises an error
   ```python
               # Accessing fragments of a filtered dataset is not supported.
               # It would be unclear if you wanted to filter the fragments
               # or the rows in those fragments.
               raise ValueError(
                   "Retrieving fragments of a filtered or projected "
                   "dataset is not allowed. Remove the filtering."
               )
   ```
   
   But it's not really relevant whether the filter is previously applied or 
passed in the call. The question is what is the right behavior if the 
expression refers to non-partition keys. A compromise proposal: allow filtering 
on only partition keys, otherwise raise an error.
   
   And if that behavior is acceptable, then the `filter` parameter in 
`get_fragments` (which pre-dates general filtering) looks redundant, and could 
be deprecated.
   
   
   ### Component(s)
   
   C++, Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to