grisaitis opened a new issue, #34554:
URL: https://github.com/apache/arrow/issues/34554

   ### Describe the usage question you have. Please include as many useful 
details as  possible.
   
   
   #### my problem
   i have a `Dataset` and want to add the `__filename` field when creating a 
`Table` from this dataset. I've tried a few things:
   
   #### attempt 1: use `dataset.scanner`
   
   ```python
   columns = dataset.schema.names + ["__filename"]  # ["__last_in_fragment"]
   scanner = dataset.scanner(columns=columns)
   my_table = scanner.to_table()
   ```
   
   but with this, i get a duplicate column error when i try to access it with 
duckdb:
   ```python
   con = duckdb.connect()
   con.execute("SELECT * FROM my_table limit 5").fetch_df()
   ```
   ```
   duckdb.InvalidInputException: Invalid Input Error: Attempting to execute an 
unsuccessful or closed pending query result
   Error: Invalid Error: ArrowInvalid: Multiple matches for 
FieldRef.Name(__filename)
   ```
   
   #### attempt 2: use a dict of columns
   i have also tried creating the new column by specifying columns as 
expressions. but, `ds.field("__filename")` is not recognized:
   
   ```python
   columns = {x: ds.field(x) for x in dataset.schema.names}
   columns["filename"] = ds.field("__filename")
   scanner = dataset.scanner(columns=columns)
   ```
   
   ```
   Traceback (most recent call last)
   ----> scanner = dataset.scanner(columns=columns)
   /opt/conda/envs/myenv/lib/python3.11/site-packages/pyarrow/_dataset.pyx:336, 
in pyarrow._dataset.Dataset.scanner()
   
/opt/conda/envs/myenv/lib/python3.11/site-packages/pyarrow/_dataset.pyx:2576, 
in pyarrow._dataset.Scanner.from_dataset()
   
/opt/conda/envs/myenv/lib/python3.11/site-packages/pyarrow/_dataset.pyx:2484, 
in pyarrow._dataset.Scanner._make_scan_options()
   
/opt/conda/envs/myenv/lib/python3.11/site-packages/pyarrow/_dataset.pyx:2381, 
in pyarrow._dataset._populate_builder()
   /opt/conda/envs/myenv/lib/python3.11/site-packages/pyarrow/error.pxi:100, in 
pyarrow.lib.check_status()
   ArrowInvalid: No match for FieldRef.Name(__filename) in <schema...>
   ```
   
   #### tl;dr
   what is the best practice for adding fields like `__filename` to a table 
while avoiding duplicate column errors?
   
   #### related issues
   
   this might be related to
   - https://github.com/apache/arrow/issues/24407
   - https://github.com/dask/dask/issues/9251
   
   
   #### versions
   
   ```
   Linux-4.19.0-23-cloud-amd64-x86_64-with-glibc2.28
   Python 3.11.0 | packaged by conda-forge (main, Jan 14 2023, 12:27:40) [GCC 
11.3.0]
   pyarrow 11.0.0
   ```
   
   
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to