HyukjinKwon commented on a change in pull request #27433:
[SPARK-30682][SPARKR][SQL] Add SparkR interface for higher order functions
URL: https://github.com/apache/spark/pull/27433#discussion_r373898606
##########
File path: R/pkg/R/functions.R
##########
@@ -3281,6 +3322,126 @@ setMethod("row_number",
###################### Collection functions######################
+#' Create o.a.s.sql.expressions.UnresolvedNamedLambdaVariable,
+#' convert it to o.s.sql.Column and wrap with R Column.
+#' Used by higher order functions.
+#'
+#' @param ... character of length = 1
+#' if length(...) > 1 then argument is interpreted as a nested
+#' Column, for example \code{unresolved_named_lambda_var("a", "b", "c")}
+#' yields unresolved \code{a.b.c}
+#' @return Column object wrapping JVM UnresolvedNamedLambdaVariable
+unresolved_named_lambda_var <- function(...) {
+ jc <- sparkR.newJObject(
+ "org.apache.spark.sql.Column",
+ sparkR.newJObject(
+
"org.apache.spark.sql.catalyst.expressions.UnresolvedNamedLambdaVariable",
+ list(...)
+ )
+ )
+ column(jc)
+}
+
+#' Create o.a.s.sql.expressions.LambdaFunction corresponding
+#' to transformation described by func.
+#' Used by higher order functions.
+#'
+#' @param fun R \code{function} (unary, binary or ternary)
+#' that transforms \code{Columns} into a \code{Column}
+#' @param expected_nargs numeric a vector of the expected
+#' number of arguments. Used for validation
+#' @return JVM \code{LambdaFunction} object
+create_lambda <- function(fun, expected_nargs) {
+ as_jexpr <- function(x) callJMethod(x@jc, "expr")
+
+ # Process function arguments
+ parameters <- formals(fun)
+ nparameters <- length(parameters)
+
+ stopifnot(nparameters %in% expected_nargs)
Review comment:
cc @felixcheung, @falaki, @shivaram, this one, I am not sure about it yet
(see https://github.com/apache/spark/pull/27406#discussion_r373896752 for more
details).
Also, given that JVM throws an error as below:
```
: org.apache.spark.sql.AnalysisException: The number of lambda function
arguments '2' does not match the number of arguments expected by the higher
order function '1'.;
```
if we use `captureJVMException` (see
https://github.com/apache/spark/pull/27433/files#r373898428), I assume it will
show R-friendly error messages.
For a simple checking in R side, I am supportive but I am less sure for such
complex checking. I tend to think it's better to have less codes if the error
message is already readable at the current moment.
We should find a better way to handle the error messages in general (e.g.,
at
[utils.R#L820-L872](https://github.com/apache/spark/blob/master/R/pkg/R/utils.R#L820-L872)).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
