[
https://issues.apache.org/jira/browse/ARROW-8245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072558#comment-17072558
]
Joris Van den Bossche commented on ARROW-8245:
----------------------------------------------
[~coverman] would you be able to do a PR this week? It would be a welcome fix
for 0.17 which is planned for the end of this week (otherwise I can also do it
this week)
There is a {{_is_private_directory}} helper function
(https://github.com/apache/arrow/blob/4269cdde067122112d810fb1a257b36d4e26163b/python/pyarrow/parquet.py#L936-L938),
which checks for underscores, and could be updated to also check for dots.
> [Python][Parquet] Skip hidden directories when reading partitioned parquet
> files
> --------------------------------------------------------------------------------
>
> Key: ARROW-8245
> URL: https://issues.apache.org/jira/browse/ARROW-8245
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Reporter: Caleb Overman
> Priority: Minor
> Labels: parquet
> Fix For: 0.17.0
>
>
> When writing a partitioned parquet file Spark can create a temporary hidden
> {{.spark-staging}} directory within the parquet file. Because it is a
> directory and not a file, it is not skipped when trying to read the parquet
> file. Pyarrow currently only skips directories prefixed with {{_}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
