[
https://issues.apache.org/jira/browse/ARROW-15060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17457285#comment-17457285
]
David Li commented on ARROW-15060:
----------------------------------
I also just tried it with Minio (which goes through S3FileSystem still) and
that seemed to work fine:
{code:java}
> minio$ls("taxi-csv-gz", recursive = TRUE)
[1] "taxi-csv-gz/2009" "taxi-csv-gz/2009/01"
[3] "taxi-csv-gz/2009/01/data.csv.gz" "taxi-csv-gz/2009/02"
[5] "taxi-csv-gz/2009/02/data.csv.gz"
> open_dataset("taxi-csv-gz", partitioning=c("year", "month"), format="csv",
> filesystem=minio)
FileSystemDataset with 2 csv files
vendor_id: string
passenger_count: int64
trip_distance: double
pickup_longitude: double
pickup_latitude: double
rate_code_id: null
store_and_fwd_flag: null
dropoff_longitude: double
dropoff_latitude: double
payment_type: string
fare_amount: double
extra: double
mta_tax: null
tip_amount: double
tolls_amount: double
total_amount: double
year: int32
month: int32 {code}
It may be worth documenting this explicitly, though, just to make sure it'll
show up in searches.
> [R] open_dataset() on csv files lacks support for compressed files
> ------------------------------------------------------------------
>
> Key: ARROW-15060
> URL: https://issues.apache.org/jira/browse/ARROW-15060
> Project: Apache Arrow
> Issue Type: Bug
> Components: R
> Reporter: Carl Boettiger
> Priority: Major
>
> Using open_dataset() on S3 buckets of csv files is a game-changing magic,
> particularly with all the additional support for database / dplyr operations
> over the remote connection, and the widespread adoption of S3 buckets even by
> old-school big data providers like NOAA.
>
> It's not uncommon to encounter buckets with *.csv.gz formats. I know
> technically this should be unnecessary, as compression can be done "in
> flight" by the server, but usually this is not an issue for R users since R's
> `connection` class automatically detects and gunzips compressed files (over
> either POSIX or HTTP connections). It would be really great if arrow could
> handle this case too.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
