orf opened a new issue, #40196:
URL: https://github.com/apache/arrow/issues/40196
### Describe the usage question you have. Please include as many useful
details as possible.
Hey,
I have a dataset that I'd like to filter using `__fragment_index` and/or
`__filename`:
```python
> from pyarrow import dataset
> ds = dataset.parquet_dataset("bucket/dataset_path/_metadata",
filesystem=S3_FS)
> ds.scanner(columns=['col_1', '__filename'],
filters=dataset.field('__fragment_index') == 1)
ArrowInvalid: No match for FieldRef.Name(__fragment_index) in col_1: string
col_2: string
col_3: string
```
The docstrings for `scanner` _imply_ that this is possible:
> The list of columns or expressions may use the special fields
> `__batch_index` (the index of the batch within the fragment),
> `__fragment_index` (the index of the fragment within the dataset),
> `__last_in_fragment` (whether the batch is last in fragment), and
> `__filename` (the name of the source file or a description of the
> source fragment).
However I can't construct a `dataset.Expression` that references any of the
special fields, and I can't see any tests that utilise this.
Am I doing something silly?
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
