[
https://issues.apache.org/jira/browse/ARROW-15280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17537068#comment-17537068
]
Neal Richardson commented on ARROW-15280:
-----------------------------------------
A couple of observations as I start to look into this:
* The code in pyarrow where this is mentioned looks to be in the legacy
ParquetDataset code path, which predates the Dataset API we use in R. I believe
this is being deprecated (right [~jorisvandenbossche]?). So it's not a drop-in
fix for R.
* There is a TODO to expose the FileSystemFactoryOptions in R, which we should
do here. That struct has a {{selector_ignore_prefixes}} option, but since this
_$folder$ is a suffix, it won't help. In fact, there are some misleading C++
tests (such as
[https://github.com/apache/arrow/blob/2e8b8365d602d8cb14a37ea4f349824549e28e8f/cpp/src/arrow/dataset/discovery_test.cc#L225-L239])
that include the _$folder$ string but not as a suffix like this. Perhaps we
should add an ignore_suffixes option, or some other interface for filtering
like this (cc [~bkietz])
* However, FileSystemFactoryOptions also has an {{exclude_invalid_files}}
option, default false, and if true should successfully exclude _$folder$ files
because they are empty. So exposing FileSystemFactoryOptions in R will allow
you to flip this and that should do the trick.
> [R] Expose FileSystemFactoryOptions
> -----------------------------------
>
> Key: ARROW-15280
> URL: https://issues.apache.org/jira/browse/ARROW-15280
> Project: Apache Arrow
> Issue Type: Improvement
> Components: R
> Affects Versions: 6.0.1
> Reporter: Bob Rudis
> Priority: Minor
> Fix For: 9.0.0
>
>
> ARROW-4406 notes that:
> >Currently reading parquet files generated by Hadoop (EMR) from S3 fails with
> >"ValueError: >Found files in an intermediate directory" because of the
> >_$folder$ empty files.
> This was fixed in the pyarrow but R still has this issue.
> The R side does not seem to have something similar to:
> {{ def _should_silently_exclude(self, file_name):}}
> {{ return (file_name.endswith('.crc') or # Checksums}}
> {{ file_name.endswith('_$folder$') or # HDFS directories in S3}}
> {{ file_name.startswith('.') or # Hidden files starting with .}}
> {{ file_name.startswith('_') or # Hidden files starting with _}}
> {{ file_name in EXCLUDED_PARQUET_PATHS)}}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
