grisaitis opened a new issue, #34554:
URL: https://github.com/apache/arrow/issues/34554
### Describe the usage question you have. Please include as many useful
details as possible.
#### my problem
i have a `Dataset` and want to add the `__filename` field when creating a
`Table` from this dataset. I've tried a few things:
#### attempt 1: use `dataset.scanner`
```python
columns = dataset.schema.names + ["__filename"] # ["__last_in_fragment"]
scanner = dataset.scanner(columns=columns)
my_table = scanner.to_table()
```
but with this, i get a duplicate column error when i try to access it with
duckdb:
```python
con = duckdb.connect()
con.execute("SELECT * FROM my_table limit 5").fetch_df()
```
```
duckdb.InvalidInputException: Invalid Input Error: Attempting to execute an
unsuccessful or closed pending query result
Error: Invalid Error: ArrowInvalid: Multiple matches for
FieldRef.Name(__filename)
```
#### attempt 2: use a dict of columns
i have also tried creating the new column by specifying columns as
expressions. but, `ds.field("__filename")` is not recognized:
```python
columns = {x: ds.field(x) for x in dataset.schema.names}
columns["filename"] = ds.field("__filename")
scanner = dataset.scanner(columns=columns)
```
```
Traceback (most recent call last)
----> scanner = dataset.scanner(columns=columns)
/opt/conda/envs/myenv/lib/python3.11/site-packages/pyarrow/_dataset.pyx:336,
in pyarrow._dataset.Dataset.scanner()
/opt/conda/envs/myenv/lib/python3.11/site-packages/pyarrow/_dataset.pyx:2576,
in pyarrow._dataset.Scanner.from_dataset()
/opt/conda/envs/myenv/lib/python3.11/site-packages/pyarrow/_dataset.pyx:2484,
in pyarrow._dataset.Scanner._make_scan_options()
/opt/conda/envs/myenv/lib/python3.11/site-packages/pyarrow/_dataset.pyx:2381,
in pyarrow._dataset._populate_builder()
/opt/conda/envs/myenv/lib/python3.11/site-packages/pyarrow/error.pxi:100, in
pyarrow.lib.check_status()
ArrowInvalid: No match for FieldRef.Name(__filename) in <schema...>
```
#### tl;dr
what is the best practice for adding fields like `__filename` to a table
while avoiding duplicate column errors?
#### related issues
this might be related to
- https://github.com/apache/arrow/issues/24407
- https://github.com/dask/dask/issues/9251
#### versions
```
Linux-4.19.0-23-cloud-amd64-x86_64-with-glibc2.28
Python 3.11.0 | packaged by conda-forge (main, Jan 14 2023, 12:27:40) [GCC
11.3.0]
pyarrow 11.0.0
```
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]