[
https://issues.apache.org/jira/browse/ARROW-15060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17460947#comment-17460947
]
Carl Boettiger commented on ARROW-15060:
----------------------------------------
My apologies, but it looks like it is read_csv_arrow over S3 which fails:
{code:java}
## arrow network csv.gz
bench::bench_time({ # ERROR
analysis <- arrow::s3_bucket(bucket = "analysis", endpoint_override =
"data.ecoforecast.org", anonymous = TRUE)
ds <-
arrow::read_csv_arrow(analysis$OpenInputFile("combined_forecasts_scores.csv.gz"))
})
{code}
The above example is on a public bucket, should be reproducible.
No problems if I use `read_csv_arrow()` locally.
Relatedly, does `write_csv_arrow()` gzip a csv if we add the ".gz" to the name?
Or is there a way to toggle compression when using `write_dataset` with
`format="csv"` ?
> [R] open_dataset() on csv files lacks support for compressed files
> ------------------------------------------------------------------
>
> Key: ARROW-15060
> URL: https://issues.apache.org/jira/browse/ARROW-15060
> Project: Apache Arrow
> Issue Type: Bug
> Components: R
> Reporter: Carl Boettiger
> Priority: Major
>
> Using open_dataset() on S3 buckets of csv files is a game-changing magic,
> particularly with all the additional support for database / dplyr operations
> over the remote connection, and the widespread adoption of S3 buckets even by
> old-school big data providers like NOAA.
>
> It's not uncommon to encounter buckets with *.csv.gz formats. I know
> technically this should be unnecessary, as compression can be done "in
> flight" by the server, but usually this is not an issue for R users since R's
> `connection` class automatically detects and gunzips compressed files (over
> either POSIX or HTTP connections). It would be really great if arrow could
> handle this case too.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
