Bob Rudis created ARROW-15280:
---------------------------------
Summary: Ignore "*_$folder$" files on S3 in {arrow} R package
Key: ARROW-15280
URL: https://issues.apache.org/jira/browse/ARROW-15280
Project: Apache Arrow
Issue Type: Improvement
Components: R
Affects Versions: 6.0.1
Reporter: Bob Rudis
ARROW-4406 notes that:
>Currently reading parquet files generated by Hadoop (EMR) from S3 fails with
>"ValueError: >Found files in an intermediate directory" because of the
>_$folder$ empty files.
This was fixed in the pyarrow but R still has this issue.
The R side does not seem to have something similar to:
{{ def _should_silently_exclude(self, file_name):}}
{{ return (file_name.endswith('.crc') or # Checksums}}
{{ file_name.endswith('_$folder$') or # HDFS directories in S3}}
{{ file_name.startswith('.') or # Hidden files starting with .}}
{{ file_name.startswith('_') or # Hidden files starting with _}}
{{ file_name in EXCLUDED_PARQUET_PATHS)}}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
