[
https://issues.apache.org/jira/browse/ARROW-8210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17067208#comment-17067208
]
Wes McKinney commented on ARROW-8210:
-------------------------------------
We should probably relax the code around duplicate column names except in
scenarios where this is definitely ambiguity.
> [C++][Dataset] Handling of duplicate columns in Dataset factory and scanning
> ----------------------------------------------------------------------------
>
> Key: ARROW-8210
> URL: https://issues.apache.org/jira/browse/ARROW-8210
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++, C++ - Dataset
> Reporter: Joris Van den Bossche
> Priority: Major
>
> While testing duplicate column names, I ran into multiple issues:
> * Factory fails if there are duplicate columns, even for a single file
> * In addition, we should also fix and/or test that factory works for
> duplicate columns if the schema's are equal
> * Once a Dataset with duplicated columns is created, scanning without any
> column projection fails
> ---
> My python reproducer:
> {code:python}
> import pyarrow as pa
> import pyarrow.parquet as pq
> import pyarrow.dataset as ds
> import pyarrow.fs
> # create single parquet file with duplicated column names
> table = pa.table([pa.array([1, 2, 3]), pa.array([4, 5, 6]), pa.array([7, 8,
> 9])], names=['a', 'b', 'a'])
> pq.write_table(table, "data_duplicate_columns.parquet")
> {code}
> Factory fails:
> {code}
> dataset = ds.dataset("data_duplicate_columns.parquet", format="parquet")
> ...
> ~/scipy/repos/arrow/python/pyarrow/dataset.py in dataset(paths_or_factories,
> filesystem, partitioning, format)
> 346
> 347 factories = [_ensure_factory(f, **kwargs) for f in
> paths_or_factories]
> --> 348 return UnionDatasetFactory(factories).finish()
> 349
> 350
> ArrowInvalid: Can't unify schema with duplicate field names.
> {code}
> And when creating a Dataset manually:
> {code:python}
> schema = pa.schema([('a', 'int64'), ('b', 'int64'), ('a', 'int64')])
> dataset = ds.FileSystemDataset(
> schema, None, ds.ParquetFileFormat(), pa.fs.LocalFileSystem(),
> [str(basedir / "data_duplicate_columns.parquet")],
> [ds.ScalarExpression(True)])
> {code}
> then scanning fails:
> {code}
> >>> dataset.to_table()
> ...
> ArrowInvalid: Multiple matches for FieldRef.Name(a) in a: int64
> b: int64
> a: int64
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
