Repository: spark Updated Branches: refs/heads/master 34ebcc6b5 -> 6424b146c
[MINOR] Update docs for functions.scala to make it clear not all the built-in functions are defined there The title summarizes the change. Author: Reynold Xin <r...@databricks.com> Closes #21318 from rxin/functions. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6424b146 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/6424b146 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/6424b146 Branch: refs/heads/master Commit: 6424b146c91fdca734a3ec972067e8e1f88e8b9e Parents: 34ebcc6 Author: Reynold Xin <r...@databricks.com> Authored: Fri Jul 27 17:24:55 2018 -0700 Committer: Xiao Li <gatorsm...@gmail.com> Committed: Fri Jul 27 17:24:55 2018 -0700 ---------------------------------------------------------------------- .../main/scala/org/apache/spark/sql/functions.scala | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/6424b146/sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---------------------------------------------------------------------- diff --git a/sql/core/src/main/scala/org/apache/spark/sql/functions.scala b/sql/core/src/main/scala/org/apache/spark/sql/functions.scala index bcd0c94..2772958 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/functions.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/functions.scala @@ -39,7 +39,21 @@ import org.apache.spark.util.Utils /** - * Functions available for DataFrame operations. + * Commonly used functions available for DataFrame operations. Using functions defined here provides + * a little bit more compile-time safety to make sure the function exists. + * + * Spark also includes more built-in functions that are less common and are not defined here. + * You can still access them (and all the functions defined here) using the `functions.expr()` API + * and calling them through a SQL expression string. You can find the entire list of functions for + * the latest version of Spark at https://spark.apache.org/docs/latest/api/sql/index.html. + * + * As an example, `isnan` is a function that is defined here. You can use `isnan(col("myCol"))` + * to invoke the `isnan` function. This way the programming language's compiler ensures `isnan` + * exists and is of the proper form. You can also use `expr("isnan(myCol)")` function to invoke the + * same function. In this case, Spark itself will ensure `isnan` exists when it analyzes the query. + * + * `regr_count` is an example of a function that is built-in but not defined here, because it is + * less commonly used. To invoke it, use `expr("regr_count(yCol, xCol)")`. * * @groupname udf_funcs UDF functions * @groupname agg_funcs Aggregate functions --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org