[
https://issues.apache.org/jira/browse/ARROW-15060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17461464#comment-17461464
]
David Li commented on ARROW-15060:
----------------------------------
Ah. Locally, we're just passing a path to {{{}read_csv_arrow{}}}, right? But
for S3, we're passing a file.
{{read_csv_arrow}} will defer to {{make_readable_file}} if given a path, which
auto-detects compression. This doesn't happen for {{{}OpenInputFile{}}}. Maybe
we can do that? (I'll defer to Jon on this.)
It doesn't seem that {{write_csv_arrow}} does any compression based on output
suffix. You would have to manually wrap it in a CompressedOutputStream, I
think. I also don't see options for compression for CSV datasets, either in R
or C++.
> [R] open_dataset() on csv files lacks support for compressed files
> ------------------------------------------------------------------
>
> Key: ARROW-15060
> URL: https://issues.apache.org/jira/browse/ARROW-15060
> Project: Apache Arrow
> Issue Type: Bug
> Components: R
> Reporter: Carl Boettiger
> Priority: Major
>
> Using open_dataset() on S3 buckets of csv files is a game-changing magic,
> particularly with all the additional support for database / dplyr operations
> over the remote connection, and the widespread adoption of S3 buckets even by
> old-school big data providers like NOAA.
>
> It's not uncommon to encounter buckets with *.csv.gz formats. I know
> technically this should be unnecessary, as compression can be done "in
> flight" by the server, but usually this is not an issue for R users since R's
> `connection` class automatically detects and gunzips compressed files (over
> either POSIX or HTTP connections). It would be really great if arrow could
> handle this case too.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
