Jonathan Keane created ARROW-13189:
--------------------------------------
Summary: [R] Should we be handling row-level metadata at all?
Key: ARROW-13189
URL: https://issues.apache.org/jira/browse/ARROW-13189
Project: Apache Arrow
Issue Type: Bug
Components: R
Affects Versions: 4.0.1, 4.0.0, 3.0.0
Reporter: Jonathan Keane
In order to support things like SF columns, we have added code that handles
row-level metadata (https://github.com/apache/arrow/pull/8549 and
https://github.com/apache/arrow/pull/9182).
These work just fine in a single table or single parquet file circumstance, but
when using a dataset (even without filtering!) this can produce some surprising
(and wrong) results (see reprex below).
There is already some work underway to make it easier to convert the
row-element-level attributes to a struct + store it in the column in the
ARROW-12542 work, but that's still a bit off. But even once that's done, should
we disable this totally? Stop or ignore+warn that with datasets row-level
metadata isn't applied (since there's no way for us to get the ordering right)?
Something else?
{code:r}
library(arrow)
df <- tibble::tibble(
part = rep(1:2, 13),
let = letters
)
df$embedded_attr <- lapply(seq_len(nrow(df)), function(i) {
value <- "nothing"
attributes(value) <- list(letter = df[[i, "let"]])
value
})
df_from_tab <- as.data.frame(Table$create(df))
# this should be (and is) "b"
attributes(df_from_tab[df_from_tab$let == "b", "embedded_attr"][[1]][[1]])
#> $letter
#> [1] "b"
# the dfs are the same
waldo::compare(df, df_from_tab)
#> ✓ No differences
# now via dataset
dir <- "ds-dir"
write_dataset(df, path = dir, partitioning = "part")
ds <- open_dataset(dir)
df_from_ds <- dplyr::collect(ds)
# this should be (and is not) "b"
attributes(df_from_ds[df_from_ds$let == "b", "embedded_attr"][[1]][[1]])
#> $letter
#> [1] "n"
# Even controlling for order, the dfs are not the same
waldo::compare(dplyr::arrange(df, let), dplyr::arrange(df_from_ds, let))
#> `names(old)`: "part" "let" "embedded_attr"
#> `names(new)`: "let" "embedded_attr" "part"
#>
#> `attr(old$embedded_attr[[2]], 'letter')`: "b"
#> `attr(new$embedded_attr[[2]], 'letter')`: "n"
#>
#> `attr(old$embedded_attr[[3]], 'letter')`: "c"
#> `attr(new$embedded_attr[[3]], 'letter')`: "b"
#>
#> `attr(old$embedded_attr[[4]], 'letter')`: "d"
#> `attr(new$embedded_attr[[4]], 'letter')`: "o"
#>
#> `attr(old$embedded_attr[[5]], 'letter')`: "e"
#> `attr(new$embedded_attr[[5]], 'letter')`: "c"
#>
#> `attr(old$embedded_attr[[6]], 'letter')`: "f"
#> `attr(new$embedded_attr[[6]], 'letter')`: "p"
#>
#> `attr(old$embedded_attr[[7]], 'letter')`: "g"
#> `attr(new$embedded_attr[[7]], 'letter')`: "d"
#>
#> `attr(old$embedded_attr[[8]], 'letter')`: "h"
#> `attr(new$embedded_attr[[8]], 'letter')`: "q"
#>
#> `attr(old$embedded_attr[[9]], 'letter')`: "i"
#> `attr(new$embedded_attr[[9]], 'letter')`: "e"
#>
#> `attr(old$embedded_attr[[10]], 'letter')`: "j"
#> `attr(new$embedded_attr[[10]], 'letter')`: "r"
#>
#> And 15 more differences ...
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
