Jonathan Keane created ARROW-11756:
--------------------------------------
Summary: [R] passing a partition as a schema leads to segfaults
Key: ARROW-11756
URL: https://issues.apache.org/jira/browse/ARROW-11756
Project: Apache Arrow
Issue Type: Bug
Components: R
Reporter: Jonathan Keane
[The command to open a dataset in
R|https://arrow.apache.org/docs/r/reference/open_dataset.html] can accept both
a schema and a partitioning argument. If one accidentally passes a partitioning
as the schema, the result looks like the dataset was read, but operating on the
dataset results in segfaults after.
Though this is input error, we should add a validation checking that the schema
argument is, in fact, a {{Schema}} object and error if it is not so that
someone doesn't find themselves confronted with a segfault later.
{code:r}
### begin setup
# note: this exact code is called in test-dataset.R lines 18-87) So when adding
# the test to that file, you don't need to copy this, but can use the code at
# the bottom of this chunk in that test if you want.
library(dplyr)
make_temp_dir <- function() {
path <- tempfile()
dir.create(path)
normalizePath(path, winslash = "/")
}
hive_dir <- make_temp_dir()
first_date <- lubridate::ymd_hms("2015-04-29 03:12:39")
df1 <- tibble(
int = 1:10,
dbl = as.numeric(1:10),
lgl = rep(c(TRUE, FALSE, NA, TRUE, FALSE), 2),
chr = letters[1:10],
fct = factor(LETTERS[1:10]),
ts = first_date + lubridate::days(1:10)
)
second_date <- lubridate::ymd_hms("2017-03-09 07:01:02")
df2 <- tibble(
int = 101:110,
dbl = c(as.numeric(51:59), NaN),
lgl = rep(c(TRUE, FALSE, NA, TRUE, FALSE), 2),
chr = letters[10:1],
fct = factor(LETTERS[10:1]),
ts = second_date + lubridate::days(10:1)
)
dir.create(file.path(hive_dir, "subdir", "group=1", "other=xxx"), recursive =
TRUE)
dir.create(file.path(hive_dir, "subdir", "group=2", "other=yyy"), recursive =
TRUE)
write_parquet(df1, file.path(hive_dir, "subdir", "group=1", "other=xxx",
"file1.parquet"))
write_parquet(df2, file.path(hive_dir, "subdir", "group=2", "other=yyy",
"file2.parquet"))
### end setup
# This (the correct specification) works just fine
ds <- open_dataset(hive_dir, partitioning = hive_partition(other = utf8(),
group = uint8()))
ds$schema
# But if you aren't explicit with ther argument names it looks like everything
works...
ds <- open_dataset(hive_dir, hive_partition(other = utf8(), group = uint8()))
# but the dataset is malformed and will have segfaults when trying to interact
with it for example:
ds$schema
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
