Matthieusalor opened a new issue, #35841:
URL: https://github.com/apache/arrow/issues/35841

   ### Describe the bug, including details regarding any error messages, 
version, and platform.
   
   Based on the documentation 
   
https://arrow.apache.org/docs/6.0/python/generated/pyarrow.dataset.Dataset.html#pyarrow.dataset.Dataset.get_fragments
   _Return fragments matching the optional filter, either using the 
partition_expression or internal information like Parquet’s statistics._
   
   I would have assumed that the following code would return only one fragment. 
However, the expression seems to be applied only to the partitions as both 
fragments are being returned even though only one matches the predicate if you 
look at the statitics
   
   ```python
   import pyarrow.dataset as pds
   import pandas as pd
   
   ds_path = './my_dataset'
   df = pd.DataFrame({
       'A': [1,2],
       'B': [1,2]
   })
   df.to_parquet(ds_path, partition_cols=['A'])
   
   pdataset = pds.dataset(ds_path, format='parquet', partitioning='hive')
   
   fragments = pdataset.get_fragments(
       filter=(pds.field('B') == 1)
   )
   
   for f in fragments:
       print('-------------')
       print(f.path)
       print(f.to_table())
       for row_group in f.row_groups:
           print(row_group.statistics)
   ```
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to