Matthieusalor opened a new issue, #35841: URL: https://github.com/apache/arrow/issues/35841
### Describe the bug, including details regarding any error messages, version, and platform. Based on the documentation https://arrow.apache.org/docs/6.0/python/generated/pyarrow.dataset.Dataset.html#pyarrow.dataset.Dataset.get_fragments _Return fragments matching the optional filter, either using the partition_expression or internal information like Parquet’s statistics._ I would have assumed that the following code would return only one fragment. However, the expression seems to be applied only to the partitions as both fragments are being returned even though only one matches the predicate if you look at the statitics ```python import pyarrow.dataset as pds import pandas as pd ds_path = './my_dataset' df = pd.DataFrame({ 'A': [1,2], 'B': [1,2] }) df.to_parquet(ds_path, partition_cols=['A']) pdataset = pds.dataset(ds_path, format='parquet', partitioning='hive') fragments = pdataset.get_fragments( filter=(pds.field('B') == 1) ) for f in fragments: print('-------------') print(f.path) print(f.to_table()) for row_group in f.row_groups: print(row_group.statistics) ``` ### Component(s) Python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
