paleolimbot commented on PR #15278:
URL: https://github.com/apache/arrow/pull/15278#issuecomment-1380860143
And verifying that this works with an exec plan collect:
Before this PR
``` r
# install.packages("arrow")
library(arrow, warn.conflicts = FALSE)
library(dplyr, warn.conflicts = FALSE)
packageVersion("arrow")
#> [1] '10.0.1'
# create a FileSystemDataset object
filename <- here::here("tmp")
write_dataset(cars, filename, format = "feather")
ds <- open_dataset(filename, format = "feather")
# Collect the dataset
cars2 <- ds %>%
filter(dist >= 2) %>%
collect()
unlink(filename, recursive = TRUE)
file.exists(filename) # should be FALSE if the fix worked
#> [1] TRUE
gc()
#> used (Mb) gc trigger (Mb) max used (Mb)
#> Ncells 1179949 63.1 2537872 135.6 1323858 70.8
#> Vcells 2042972 15.6 8388608 64.0 3637395 27.8
unlink(filename, recursive = TRUE)
file.exists(filename)
#> [1] FALSE
```
<sup>Created on 2023-01-12 with [reprex
v2.0.2](https://reprex.tidyverse.org)</sup>
After this PR
``` r
library(arrow, warn.conflicts = FALSE)
library(dplyr, warn.conflicts = FALSE)
packageVersion("arrow")
#> [1] '10.0.1.100000401'
# create a FileSystemDataset object
filename <- here::here("tmp")
write_dataset(cars, filename, format = "feather")
ds <- open_dataset(filename, format = "feather")
# Collect the dataset
cars2 <- ds %>%
filter(dist >= 2) %>%
collect()
unlink(filename, recursive = TRUE)
file.exists(filename) # should be FALSE if the fix worked
#> [1] FALSE
```
<sup>Created on 2023-01-12 with [reprex
v2.0.2](https://reprex.tidyverse.org)</sup>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
