[
https://issues.apache.org/jira/browse/ARROW-8446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joris Van den Bossche updated ARROW-8446:
-----------------------------------------
Labels: dataset dataset-dask-integration parquet (was: dataset parquet)
> [Python][Dataset] Detect and use _metadata file in a list of file paths
> -----------------------------------------------------------------------
>
> Key: ARROW-8446
> URL: https://issues.apache.org/jira/browse/ARROW-8446
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Reporter: Joris Van den Bossche
> Priority: Major
> Labels: dataset, dataset-dask-integration, parquet
>
> From https://github.com/dask/dask/pull/6047#discussion_r402391318
> When specifying a directory to {{ParquetDataset}}, we will detect if a
> {{_metadata}} file is present in the directory and use that to populate the
> {{metadata}} attribute (and not include this file in the list of "pieces",
> since it does not include any data).
>
> However, when passing a list of files to {{ParquetDataset}}, with one being
> "_metadata", the metadata attribute is not populated, and the "_metadata"
> path is included as one of the ParquetDatasetPiece objects instead (which
> leads to an ArrowIOError during the read of that piece).
> We _could_ detect it in a list of paths as well.
> Note, I mentioned {{ParquetDataset}}, but if working on this, we should
> probably directly do it in the datasets API-based version.
> Also, I labeled this as Python and not C++ for now, as this might be
> something that can be handled on the Python side (once the C++ side knows how
> to process this kind of metadata -> ARROW-8062)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
