ablack3 opened a new issue, #37758:
URL: https://github.com/apache/arrow/issues/37758
### Describe the bug, including details regarding any error messages,
version, and platform.
Some dplyr functions use the tbl_vars generic that does not seem to be
implemented for FileSystemDataset objects yet.
``` r
library(arrow)
#>
#> Attaching package: 'arrow'
#> The following object is masked from 'package:utils':
#>
#> timestamp
write_dataset(cars, file.path(tempdir(), "cars.parquet"))
df <- open_dataset(file.path(tempdir(), "cars.parquet"))
library(dplyr, warn.conflicts = F)
# this works
df %>%
mutate(across(is.numeric, list(newcolumn = ~.x*10))) %>%
collect()
#> Warning: Use of bare predicate functions was deprecated in tidyselect
1.1.0.
#> ℹ Please use wrap predicates in `where()` instead.
#> # Was:
#> data %>% select(is.numeric)
#>
#> # Now:
#> data %>% select(where(is.numeric))
#> This warning is displayed once every 8 hours.
#> Call `lifecycle::last_lifecycle_warnings()` to see where this warning was
#> generated.
#> # A tibble: 50 × 4
#> speed dist speed_newcolumn dist_newcolumn
#> <dbl> <dbl> <dbl> <dbl>
#> 1 4 2 40 20
#> 2 4 10 40 100
#> 3 7 4 70 40
#> 4 7 22 70 220
#> 5 8 16 80 160
#> 6 9 10 90 100
#> 7 10 18 100 180
#> 8 10 26 100 260
#> 9 10 34 100 340
#> 10 11 17 110 170
#> # ℹ 40 more rows
# these don't
df %>%
mutate_if(is.numeric, ~.x*10) %>%
collect()
#> Error in UseMethod("tbl_vars"): no applicable method for 'tbl_vars'
applied to an object of class "c('FileSystemDataset', 'Dataset', 'ArrowObject',
'R6')"
df %>%
mutate(across(is.numeric, list(newcolumn = function(x) x*10))) %>%
collect()
#> Error in `across_setup()`:
#> ! Anonymous functions are not yet supported in Arrow
#> Backtrace:
#> ▆
#> 1. ├─... %>% collect()
#> 2. ├─dplyr::collect(.)
#> 3. ├─dplyr::mutate(...)
#> 4. └─arrow:::mutate.Dataset(...)
#> 5. └─arrow:::expand_across(out, quos(...), exclude_cols = grv)
#> 6. └─arrow:::across_setup(...)
#> 7. └─rlang::abort("Anonymous functions are not yet supported in
Arrow")
df %>%
mutate_at(vars(speed), ~.x*10) %>%
collect()
#> Error in UseMethod("tbl_vars"): no applicable method for 'tbl_vars'
applied to an object of class "c('FileSystemDataset', 'Dataset', 'ArrowObject',
'R6')"
# this does because we pull data into R first
df %>%
collect() %>%
mutate_at(vars(speed), ~.x*10)
#> # A tibble: 50 × 2
#> speed dist
#> <dbl> <dbl>
#> 1 40 2
#> 2 40 10
#> 3 70 4
#> 4 70 22
#> 5 80 16
#> 6 90 10
#> 7 100 18
#> 8 100 26
#> 9 100 34
#> 10 110 17
#> # ℹ 40 more rows
sessionInfo()
#> R version 4.3.1 (2023-06-16)
#> Platform: aarch64-apple-darwin20 (64-bit)
#> Running under: macOS Ventura 13.5.2
#>
#> Matrix products: default
#> BLAS:
/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libRblas.0.dylib
#> LAPACK:
/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libRlapack.dylib;
LAPACK version 3.11.0
#>
#> locale:
#> [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
#>
#> time zone: America/Chicago
#> tzcode source: internal
#>
#> attached base packages:
#> [1] stats graphics grDevices utils datasets methods base
#>
#> other attached packages:
#> [1] dplyr_1.1.3 arrow_13.0.0
#>
#> loaded via a namespace (and not attached):
#> [1] vctrs_0.6.3 cli_3.6.1 knitr_1.44 rlang_1.1.1
#> [5] xfun_0.40 purrr_1.0.2 generics_0.1.3
assertthat_0.2.1
#> [9] glue_1.6.2 bit_4.0.5 htmltools_0.5.6 fansi_1.0.4
#> [13] rmarkdown_2.24 tibble_3.2.1 evaluate_0.21 fastmap_1.1.1
#> [17] yaml_2.3.7 lifecycle_1.0.3 compiler_4.3.1 fs_1.6.3
#> [21] pkgconfig_2.0.3 rstudioapi_0.15.0 digest_0.6.33 R6_2.5.1
#> [25] utf8_1.2.3 reprex_2.0.2 tidyselect_1.2.0 pillar_1.9.0
#> [29] magrittr_2.0.3 tools_4.3.1 withr_2.5.0 bit64_4.0.5
```
<sup>Created on 2023-09-17 with [reprex
v2.0.2](https://reprex.tidyverse.org)</sup>
### Component(s)
R
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
