[
https://issues.apache.org/jira/browse/ARROW-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Culshaw-Maurer updated ARROW-16598:
-------------------------------------------
Description:
When using the arrow R package, sorting a data.frame prior to using
write_parquet() results in different file sizes, depending on how the
data.frame is sorted. I have attached a reproducible example showing how a few
different sorting methods can lead to 2-3 fold changes in .parquet file size.
It may be that I don't know enough about Parquet internals, but at the very
least, I think this behavior should be documented on the arrow R package site.
Most R users tend to approach sorting as a convenience and don't expect it to
lead to performance changes when writing to a file.
```
library(tidyverse)
d <- expand_grid(group = letters[1:4],
id = 1:100) %>%
mutate(id_f = paste(group, id, sep = "_")) %>%
mutate(time = rep(list(1:100)), 400) %>%
unnest(time) %>%
group_by(group) %>%
mutate(id_n = list(sample(id_f, size = 5, replace = F))) %>%
unnest(id_n) %>%
ungroup()
f1 <- tempfile(fileext = ".parquet")
f2 <- tempfile(fileext = ".parquet")
f3 <- tempfile(fileext = ".parquet")
f4 <- tempfile(fileext = ".parquet")
f5 <- tempfile(fileext = ".parquet")
d %>%
arrow::write_parquet(f1)
d %>%
arrange(id_n) %>%
arrow::write_parquet(f2)
d %>%
arrange(id_n, time) %>%
arrow::write_parquet(f3)
d %>%
arrange(time, id_f) %>%
arrow::write_parquet(f4)
d %>%
arrange(group, time, id_n, id_f) %>%
arrow::write_parquet(f5)
fs::file_info(c(f1, f2, f3, f4, f5))[, "size"]
#> # A tibble: 5 × 1
#> size
#> <fs::bytes>
#> 1 25.4K
#> 2 17.3K
#> 3 28.4K
#> 4 45.6K
#> 5 30.1K
sessioninfo::session_info()
#> ─ Session info
───────────────────────────────────────────────────────────────
#> setting value
#> version R version 4.1.3 (2022-03-10)
#> os macOS Big Sur/Monterey 10.16
#> system x86_64, darwin17.0
#> ui X11
#> language (EN)
#> collate en_US.UTF-8
#> ctype en_US.UTF-8
#> tz America/Chicago
#> date 2022-05-17
#>
#> ─ Packages
───────────────────────────────────────────────────────────────────
#> package * version date lib source
#> arrow 8.0.0 2022-05-09 [1] CRAN (R 4.1.2)
#> assertthat 0.2.1 2019-03-21 [1] CRAN (R 4.1.0)
#> backports 1.4.1 2021-12-13 [1] CRAN (R 4.1.0)
#> bit 4.0.4 2020-08-04 [1] CRAN (R 4.1.0)
#> bit64 4.0.5 2020-08-30 [1] CRAN (R 4.1.0)
#> broom 0.7.9 2021-07-27 [1] CRAN (R 4.1.0)
#> cellranger 1.1.0 2016-07-27 [1] CRAN (R 4.1.0)
#> cli 3.3.0 2022-04-25 [1] CRAN (R 4.1.2)
#> colorspace 2.0-3 2022-02-21 [1] CRAN (R 4.1.2)
#> crayon 1.5.1 2022-03-26 [1] CRAN (R 4.1.2)
#> DBI 1.1.1 2021-01-15 [1] CRAN (R 4.1.0)
#> dbplyr 2.1.1 2021-04-06 [1] CRAN (R 4.1.0)
#> digest 0.6.29 2021-12-01 [1] CRAN (R 4.1.0)
#> dplyr * 1.0.9 2022-04-28 [1] CRAN (R 4.1.2)
#> ellipsis 0.3.2 2021-04-29 [1] CRAN (R 4.1.0)
#> evaluate 0.14 2019-05-28 [1] CRAN (R 4.1.0)
#> fansi 1.0.3 2022-03-24 [1] CRAN (R 4.1.2)
#> fastmap 1.1.0 2021-01-25 [1] CRAN (R 4.1.0)
#> forcats * 0.5.1 2021-01-27 [1] CRAN (R 4.1.0)
#> fs 1.5.2 2021-12-08 [1] CRAN (R 4.1.0)
#> generics 0.1.2 2022-01-31 [1] CRAN (R 4.1.2)
#> ggplot2 * 3.3.6 2022-05-03 [1] CRAN (R 4.1.3)
#> glue 1.6.2 2022-02-24 [1] CRAN (R 4.1.2)
#> gtable 0.3.0 2019-03-25 [1] CRAN (R 4.1.0)
#> haven 2.4.3 2021-08-04 [1] CRAN (R 4.1.0)
#> highr 0.9 2021-04-16 [1] CRAN (R 4.1.0)
#> hms 1.1.0 2021-05-17 [1] CRAN (R 4.1.0)
#> htmltools 0.5.2 2021-08-25 [1] CRAN (R 4.1.0)
#> httr 1.4.2 2020-07-20 [1] CRAN (R 4.1.0)
#> jsonlite 1.8.0 2022-02-22 [1] CRAN (R 4.1.2)
#> knitr 1.37 2021-12-16 [1] CRAN (R 4.1.0)
#> lifecycle 1.0.1 2021-09-24 [1] CRAN (R 4.1.0)
#> lubridate 1.7.10 2021-02-26 [1] CRAN (R 4.1.0)
#> magrittr 2.0.3 2022-03-30 [1] CRAN (R 4.1.2)
#> modelr 0.1.8 2020-05-19 [1] CRAN (R 4.1.0)
#> munsell 0.5.0 2018-06-12 [1] CRAN (R 4.1.0)
#> pillar 1.7.0 2022-02-01 [1] CRAN (R 4.1.2)
#> pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.1.0)
#> purrr * 0.3.4 2020-04-17 [1] CRAN (R 4.1.0)
#> R6 2.5.1 2021-08-19 [1] CRAN (R 4.1.0)
#> Rcpp 1.0.8.3 2022-03-17 [1] CRAN (R 4.1.2)
#> readr * 2.0.1 2021-08-10 [1] CRAN (R 4.1.0)
#> readxl 1.3.1 2019-03-13 [1] CRAN (R 4.1.0)
#> reprex 2.0.1 2021-08-05 [1] CRAN (R 4.1.0)
#> rlang 1.0.2 2022-03-04 [1] CRAN (R 4.1.2)
#> rmarkdown 2.11 2021-09-14 [1] CRAN (R 4.1.0)
#> rstudioapi 0.13 2020-11-12 [1] CRAN (R 4.1.0)
#> rvest 1.0.1 2021-07-26 [1] CRAN (R 4.1.0)
#> scales 1.2.0 2022-04-13 [1] CRAN (R 4.1.2)
#> sessioninfo 1.1.1 2018-11-05 [1] CRAN (R 4.1.0)
#> stringi 1.7.6 2021-11-29 [1] CRAN (R 4.1.0)
#> stringr * 1.4.0 2019-02-10 [1] CRAN (R 4.1.0)
#> styler 1.4.1 2021-03-30 [1] CRAN (R 4.1.0)
#> tibble * 3.1.7 2022-05-03 [1] CRAN (R 4.1.3)
#> tidyr * 1.1.3 2021-03-03 [1] CRAN (R 4.1.0)
#> tidyselect 1.1.2 2022-02-21 [1] CRAN (R 4.1.2)
#> tidyverse * 1.3.1 2021-04-15 [1] CRAN (R 4.1.0)
#> tzdb 0.1.2 2021-07-20 [1] CRAN (R 4.1.0)
#> utf8 1.2.2 2021-07-24 [1] CRAN (R 4.1.0)
#> vctrs 0.4.1 2022-04-13 [1] CRAN (R 4.1.2)
#> withr 2.5.0 2022-03-03 [1] CRAN (R 4.1.2)
#> xfun 0.30 2022-03-02 [1] CRAN (R 4.1.2)
#> xml2 1.3.2 2020-04-23 [1] CRAN (R 4.1.0)
#> yaml 2.3.5 2022-02-21 [1] CRAN (R 4.1.2)
#>
#> [1] /Users/MJ/R_Packages_4.1
#> [2] /Library/Frameworks/R.framework/Versions/4.1/Resources/library
```
was:
When using the arrow R package, sorting a data.frame prior to using
write_parquet() results in different file sizes, depending on how the
data.frame is sorted. I have attached a reproducible example showing how a few
different sorting methods can lead to 2-3 fold changes in .parquet file size.
It may be that I don't know enough about Parquet internals, but at the very
least, I think this behavior should be documented on the arrow R package site.
Most R users tend to approach sorting as a convenience and don't expect it to
lead to performance changes when writing to a file.
> [R] Sorting data.frame prior to writing Parquet affects file size
> -----------------------------------------------------------------
>
> Key: ARROW-16598
> URL: https://issues.apache.org/jira/browse/ARROW-16598
> Project: Apache Arrow
> Issue Type: Bug
> Components: R
> Environment: MacBook Pro (non-M1), other info in R file
> Reporter: Michael Culshaw-Maurer
> Priority: Minor
> Attachments: arrow_parquet_bug.R
>
>
> When using the arrow R package, sorting a data.frame prior to using
> write_parquet() results in different file sizes, depending on how the
> data.frame is sorted. I have attached a reproducible example showing how a
> few different sorting methods can lead to 2-3 fold changes in .parquet file
> size.
> It may be that I don't know enough about Parquet internals, but at the very
> least, I think this behavior should be documented on the arrow R package
> site. Most R users tend to approach sorting as a convenience and don't expect
> it to lead to performance changes when writing to a file.
>
> ```
> library(tidyverse)
> d <- expand_grid(group = letters[1:4],
> id = 1:100) %>%
> mutate(id_f = paste(group, id, sep = "_")) %>%
> mutate(time = rep(list(1:100)), 400) %>%
> unnest(time) %>%
> group_by(group) %>%
> mutate(id_n = list(sample(id_f, size = 5, replace = F))) %>%
> unnest(id_n) %>%
> ungroup()
> f1 <- tempfile(fileext = ".parquet")
> f2 <- tempfile(fileext = ".parquet")
> f3 <- tempfile(fileext = ".parquet")
> f4 <- tempfile(fileext = ".parquet")
> f5 <- tempfile(fileext = ".parquet")
> d %>%
> arrow::write_parquet(f1)
> d %>%
> arrange(id_n) %>%
> arrow::write_parquet(f2)
> d %>%
> arrange(id_n, time) %>%
> arrow::write_parquet(f3)
> d %>%
> arrange(time, id_f) %>%
> arrow::write_parquet(f4)
> d %>%
> arrange(group, time, id_n, id_f) %>%
> arrow::write_parquet(f5)
> fs::file_info(c(f1, f2, f3, f4, f5))[, "size"]
> #> # A tibble: 5 × 1
> #> size
> #> <fs::bytes>
> #> 1 25.4K
> #> 2 17.3K
> #> 3 28.4K
> #> 4 45.6K
> #> 5 30.1K
> sessioninfo::session_info()
> #> ─ Session info
> ───────────────────────────────────────────────────────────────
> #> setting value
> #> version R version 4.1.3 (2022-03-10)
> #> os macOS Big Sur/Monterey 10.16
> #> system x86_64, darwin17.0
> #> ui X11
> #> language (EN)
> #> collate en_US.UTF-8
> #> ctype en_US.UTF-8
> #> tz America/Chicago
> #> date 2022-05-17
> #>
> #> ─ Packages
> ───────────────────────────────────────────────────────────────────
> #> package * version date lib source
> #> arrow 8.0.0 2022-05-09 [1] CRAN (R 4.1.2)
> #> assertthat 0.2.1 2019-03-21 [1] CRAN (R 4.1.0)
> #> backports 1.4.1 2021-12-13 [1] CRAN (R 4.1.0)
> #> bit 4.0.4 2020-08-04 [1] CRAN (R 4.1.0)
> #> bit64 4.0.5 2020-08-30 [1] CRAN (R 4.1.0)
> #> broom 0.7.9 2021-07-27 [1] CRAN (R 4.1.0)
> #> cellranger 1.1.0 2016-07-27 [1] CRAN (R 4.1.0)
> #> cli 3.3.0 2022-04-25 [1] CRAN (R 4.1.2)
> #> colorspace 2.0-3 2022-02-21 [1] CRAN (R 4.1.2)
> #> crayon 1.5.1 2022-03-26 [1] CRAN (R 4.1.2)
> #> DBI 1.1.1 2021-01-15 [1] CRAN (R 4.1.0)
> #> dbplyr 2.1.1 2021-04-06 [1] CRAN (R 4.1.0)
> #> digest 0.6.29 2021-12-01 [1] CRAN (R 4.1.0)
> #> dplyr * 1.0.9 2022-04-28 [1] CRAN (R 4.1.2)
> #> ellipsis 0.3.2 2021-04-29 [1] CRAN (R 4.1.0)
> #> evaluate 0.14 2019-05-28 [1] CRAN (R 4.1.0)
> #> fansi 1.0.3 2022-03-24 [1] CRAN (R 4.1.2)
> #> fastmap 1.1.0 2021-01-25 [1] CRAN (R 4.1.0)
> #> forcats * 0.5.1 2021-01-27 [1] CRAN (R 4.1.0)
> #> fs 1.5.2 2021-12-08 [1] CRAN (R 4.1.0)
> #> generics 0.1.2 2022-01-31 [1] CRAN (R 4.1.2)
> #> ggplot2 * 3.3.6 2022-05-03 [1] CRAN (R 4.1.3)
> #> glue 1.6.2 2022-02-24 [1] CRAN (R 4.1.2)
> #> gtable 0.3.0 2019-03-25 [1] CRAN (R 4.1.0)
> #> haven 2.4.3 2021-08-04 [1] CRAN (R 4.1.0)
> #> highr 0.9 2021-04-16 [1] CRAN (R 4.1.0)
> #> hms 1.1.0 2021-05-17 [1] CRAN (R 4.1.0)
> #> htmltools 0.5.2 2021-08-25 [1] CRAN (R 4.1.0)
> #> httr 1.4.2 2020-07-20 [1] CRAN (R 4.1.0)
> #> jsonlite 1.8.0 2022-02-22 [1] CRAN (R 4.1.2)
> #> knitr 1.37 2021-12-16 [1] CRAN (R 4.1.0)
> #> lifecycle 1.0.1 2021-09-24 [1] CRAN (R 4.1.0)
> #> lubridate 1.7.10 2021-02-26 [1] CRAN (R 4.1.0)
> #> magrittr 2.0.3 2022-03-30 [1] CRAN (R 4.1.2)
> #> modelr 0.1.8 2020-05-19 [1] CRAN (R 4.1.0)
> #> munsell 0.5.0 2018-06-12 [1] CRAN (R 4.1.0)
> #> pillar 1.7.0 2022-02-01 [1] CRAN (R 4.1.2)
> #> pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.1.0)
> #> purrr * 0.3.4 2020-04-17 [1] CRAN (R 4.1.0)
> #> R6 2.5.1 2021-08-19 [1] CRAN (R 4.1.0)
> #> Rcpp 1.0.8.3 2022-03-17 [1] CRAN (R 4.1.2)
> #> readr * 2.0.1 2021-08-10 [1] CRAN (R 4.1.0)
> #> readxl 1.3.1 2019-03-13 [1] CRAN (R 4.1.0)
> #> reprex 2.0.1 2021-08-05 [1] CRAN (R 4.1.0)
> #> rlang 1.0.2 2022-03-04 [1] CRAN (R 4.1.2)
> #> rmarkdown 2.11 2021-09-14 [1] CRAN (R 4.1.0)
> #> rstudioapi 0.13 2020-11-12 [1] CRAN (R 4.1.0)
> #> rvest 1.0.1 2021-07-26 [1] CRAN (R 4.1.0)
> #> scales 1.2.0 2022-04-13 [1] CRAN (R 4.1.2)
> #> sessioninfo 1.1.1 2018-11-05 [1] CRAN (R 4.1.0)
> #> stringi 1.7.6 2021-11-29 [1] CRAN (R 4.1.0)
> #> stringr * 1.4.0 2019-02-10 [1] CRAN (R 4.1.0)
> #> styler 1.4.1 2021-03-30 [1] CRAN (R 4.1.0)
> #> tibble * 3.1.7 2022-05-03 [1] CRAN (R 4.1.3)
> #> tidyr * 1.1.3 2021-03-03 [1] CRAN (R 4.1.0)
> #> tidyselect 1.1.2 2022-02-21 [1] CRAN (R 4.1.2)
> #> tidyverse * 1.3.1 2021-04-15 [1] CRAN (R 4.1.0)
> #> tzdb 0.1.2 2021-07-20 [1] CRAN (R 4.1.0)
> #> utf8 1.2.2 2021-07-24 [1] CRAN (R 4.1.0)
> #> vctrs 0.4.1 2022-04-13 [1] CRAN (R 4.1.2)
> #> withr 2.5.0 2022-03-03 [1] CRAN (R 4.1.2)
> #> xfun 0.30 2022-03-02 [1] CRAN (R 4.1.2)
> #> xml2 1.3.2 2020-04-23 [1] CRAN (R 4.1.0)
> #> yaml 2.3.5 2022-02-21 [1] CRAN (R 4.1.2)
> #>
> #> [1] /Users/MJ/R_Packages_4.1
> #> [2] /Library/Frameworks/R.framework/Versions/4.1/Resources/library
> ```
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
