[
https://issues.apache.org/jira/browse/SPARK-9972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14699237#comment-14699237
]
Yu Ishikawa commented on SPARK-9972:
------------------------------------
This is a quick note to tell the reason. When I tried to implement
{{sort_array}}, I got the error as follows. I haven't inspected it, but the
cause seems to be at {{collect}}. I'll comment about that in detail later.
{noformat}
1. Error: sort_array on a DataFrame --------------------------------------------
cannot coerce class ""jobj"" to a data.frame
1: withCallingHandlers(eval(code, new_test_environment), error = capture_calls,
message = function(c) invokeRestart("muffleMessage"),
warning = function(c) invokeRestart("muffleWarning"))
2: eval(code, new_test_environment)
3: eval(expr, envir, enclos)
4: expect_equal(collect(select(df, sort_array(df$a)))[1, 1], c(1, 2, 3)) at
test_sparkSQL.R:787
5: expect_that(object, equals(expected, label = expected.label, ...), info =
info, label = label)
6: condition(object)
7: compare(expected, actual, ...)
8: compare.numeric(expected, actual, ...)
9: all.equal(x, y, ...)
10: all.equal.numeric(x, y, ...)
11: attr.all.equal(target, current, tolerance = tolerance, scale = scale, ...)
12: mode(current)
13: collect(select(df, sort_array(df$a)))
14: collect(select(df, sort_array(df$a)))
15: .local(x, ...)
16: do.call(cbind.data.frame, list(cols, stringsAsFactors = stringsAsFactors))
17: (function (..., deparse.level = 1)
data.frame(..., check.names = FALSE))(structure(list(`sort_array(a,true)` =
list(
<environment>, NA, NA)), .Names = "sort_array(a,true)"),
stringsAsFactors = FALSE)
18: data.frame(..., check.names = FALSE)
19: as.data.frame(x[[i]], optional = TRUE, stringsAsFactors = stringsAsFactors)
20: as.data.frame.list(x[[i]], optional = TRUE, stringsAsFactors =
stringsAsFactors)
21: eval(as.call(c(expression(data.frame), x, check.names = !optional,
stringsAsFactors = stringsAsFactors)))
22: eval(expr, envir, enclos)
23: data.frame(`sort_array(a,true)` = list(<environment>, NA, NA), check.names
= FALSE,
stringsAsFactors = FALSE)
24: as.data.frame(x[[i]], optional = TRUE, stringsAsFactors = stringsAsFactors)
25: as.data.frame.list(x[[i]], optional = TRUE, stringsAsFactors =
stringsAsFactors)
26: eval(as.call(c(expression(data.frame), x, check.names = !optional,
stringsAsFactors = stringsAsFactors)))
27: eval(expr, envir, enclos)
28: data.frame(<environment>, NA, NA, check.names = FALSE, stringsAsFactors =
FALSE)
29: as.data.frame(x[[i]], optional = TRUE)
30: as.data.frame.default(x[[i]], optional = TRUE)
31: stop(gettextf("cannot coerce class \"%s\" to a data.frame",
deparse(class(x))), domain = NA)
32: .handleSimpleError(function (e)
{
e$calls <- head(sys.calls()[-seq_len(frame + 7)], -2)
signalCondition(e)
}, "cannot coerce class \"\"jobj\"\" to a data.frame",
quote(as.data.frame.default(x[[i]],
optional = TRUE)))
{noformat}
> Add `struct`, `encode` and `decode` function in SparkR
> ------------------------------------------------------
>
> Key: SPARK-9972
> URL: https://issues.apache.org/jira/browse/SPARK-9972
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
> Reporter: Yu Ishikawa
>
> Support {{struct}} function on a DataFrame in SparkR. However, I think we
> need to improve {{collect}} function in SparkR in order to implement
> {{struct}} function.
> - struct
> - encode
> - decode
> - array_contains
> - sort_array
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
