nealrichardson commented on a change in pull request #10190:
URL: https://github.com/apache/arrow/pull/10190#discussion_r628588157
##########
File path: r/tests/testthat/test-dplyr-string-functions.R
##########
@@ -239,7 +254,172 @@ test_that("str_replace and str_replace_all", {
})
-test_that("backreferences in pattern", {
+test_that("strsplit and str_split", {
+
+ df <- tibble(x = c("Foo and bar", "baz and qux and quux"))
+
+ expect_dplyr_equal(
+ input %>%
+ mutate(x = strsplit(x, "and")) %>%
+ collect(),
+ df
+ )
+ expect_dplyr_equal(
+ input %>%
+ mutate(x = strsplit(x, "and.*", fixed = TRUE)) %>%
+ collect(),
+ df
+ )
+ expect_dplyr_equal(
+ input %>%
+ mutate(x = str_split(x, "and")) %>%
+ collect(),
+ df
+ )
+ expect_dplyr_equal(
+ input %>%
+ mutate(x = str_split(x, "and", n = 2)) %>%
+ collect(),
+ df
+ )
+ expect_dplyr_equal(
+ input %>%
+ mutate(x = str_split(x, fixed("and"), n = 2)) %>%
+ collect(),
+ df
+ )
+ expect_dplyr_equal(
+ input %>%
+ mutate(x = str_split(x, regex("and"), n = 2)) %>%
+ collect(),
+ df
+ )
+
+})
+
+test_that("arrow_*_split_whitespace functions", {
+
+ # use only ASCII whitespace characters
+ df_ascii <- tibble(x = c("Foo\nand bar", "baz\tand qux and quux"))
+
+ # use only non-ASCII whitespace characters
+ df_utf8 <- tibble(x = c("Foo\u00A0and\u2000bar",
"baz\u2006and\u1680qux\u3000and\u2008quux"))
+
+ df_split <- tibble(x = list(c("Foo", "and", "bar"), c("baz", "and", "qux",
"and", "quux")))
+
+ expect_equivalent(
+ df_ascii %>%
+ Table$create() %>%
+ mutate(x = arrow_ascii_split_whitespace(x)) %>%
+ collect(),
+ df_split
+ )
+ expect_equivalent(
+ df_utf8 %>%
+ Table$create() %>%
+ mutate(x = arrow_utf8_split_whitespace(x)) %>%
+ collect(),
+ df_split
+ )
+
+})
+
+test_that("errors and warnings in string splitting", {
+ df <- tibble(x = c("Foo and bar", "baz and qux and quux"))
+
+ # These conditions generate an error, but abandon_ship() catches the error,
+ # issues a warning, and pulls the data into R
+ expect_warning(
+ df %>%
+ Table$create() %>%
+ mutate(x = strsplit(x, "and.*", fixed = FALSE)) %>%
+ collect(),
+ regexp = "not supported"
Review comment:
I thought we were already on a separate JIRA at this point :)
Sure, another JIRA is fine. My eyes were drawn to it here because there was
no assertion about the result of the dplyr pipeline, just that it raised a
warning, whereas I feel like in other places we still `expect_dplyr_equal` or
whatever (which, to the point about this being a global issue, is probably
unnecessary once we've established somewhere that `abandon_ship` works).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
