[
https://issues.apache.org/jira/browse/ARROW-8245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Caleb Overman updated ARROW-8245:
---------------------------------
Summary: [Python][Parquet] Skip hidden directories when reading partitioned
parquet files (was: [Python] Skip hidden directories when reading partitioned
parquet files)
> [Python][Parquet] Skip hidden directories when reading partitioned parquet
> files
> --------------------------------------------------------------------------------
>
> Key: ARROW-8245
> URL: https://issues.apache.org/jira/browse/ARROW-8245
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Reporter: Caleb Overman
> Priority: Minor
> Labels: parquet
>
> When writing a partitioned parquet file Spark can create a temporary hidden
> `.spark-staging` directory within the parquet file. Because it is a directory
> and not a file, it is not skipped when trying to read the parquet file.
> Pyarrow currently only skips directories prefixed with `_`.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
