paleolimbot commented on issue #33784:
URL: https://github.com/apache/arrow/issues/33784#issuecomment-1440174266
Echoing Nic's thanks for opening this...our support for list columns is far
from perfect. In particular, we drop names without warning, which we should fix.
Supporting names internally in Arrow is hard because Arrow doesn't have an
internal concept of named things so we would have to invent one. We will
probably get there - probably via an extension type - but in the meantime you
will have to do some conversion to/from arrow yourself as a workaround.
The two workarounds I can think of off the top of my head are (1) serialize
list objects on the way in and unserialize them on the way out:
``` r
library(tibble)
tb <- tibble(list_column = list(c(a = 1, b = 2)))
str(tb$list_column)
#> List of 1
#> $ : Named num [1:2] 1 2
#> ..- attr(*, "names")= chr [1:2] "a" "b"
serialize_list_col_to_binary <- function(x) {
lapply(x, serialize, NULL)
}
unserialize_list_col_from_binary <- function(x) {
lapply(x, unserialize)
}
# Write + read back:
tmpf <- tempfile()
tb$list_column <- serialize_list_col_to_binary(tb$list_column)
arrow::write_feather(tb, tmpf)
df2 <- arrow::read_feather(tmpf)
df2$list_column <- unserialize_list_col_from_binary(df2$list_column)
str(df2$list_column)
#> List of 1
#> $ : Named num [1:2] 1 2
#> ..- attr(*, "names")= chr [1:2] "a" "b"
```
...or (2) do some of your own modifications to make the list element types
fit better in Arrow. In your case, your list elements could be data.frames:
``` r
library(tibble)
tb <- tibble(list_column = list(c(a = 1, b = 2)))
str(tb$list_column)
#> List of 1
#> $ : Named num [1:2] 1 2
#> ..- attr(*, "names")= chr [1:2] "a" "b"
list_col_to_arrow_friendly <- function(x) {
lapply(x, function(x) {
if (is.null(x)) NULL else as.data.frame(as.list(x))
})
}
tb$list_column <- list_col_to_arrow_friendly(tb$list_column)
str(tb$list_column)
#> List of 1
#> $ :'data.frame': 1 obs. of 2 variables:
#> ..$ a: num 1
#> ..$ b: num 2
# Write + read back:
tmpf <- tempfile()
arrow::write_feather(tb, tmpf)
df2 <- arrow::read_feather(tmpf)
str(df2$list_column)
#> list<
#> tbl_df<
#> a: double
#> b: double
#> >
#> > [1:1]
#> $ : tibble [1 × 2] (S3: tbl_df/tbl/data.frame)
#> ..$ a: num 1
#> ..$ b: num 2
#> @ ptype: tibble [0 × 2] (S3: tbl_df/tbl/data.frame)
#> ..$ a: num(0)
#> ..$ b: num(0)
```
<sup>Created on 2023-02-22 with [reprex
v2.0.2](https://reprex.tidyverse.org)</sup>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
