[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon closed the pull request at: https://github.com/apache/spark/pull/15513 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r85624958 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/HyperLogLogPlusPlus.scala --- @@ -47,10 +47,16 @@ import org.apache.spark.sql.types._ */ // scalastyle:on @ExpressionDescription( - usage = """_FUNC_(expr) - Returns the estimated cardinality by HyperLogLog++. -_FUNC_(expr, relativeSD=0.05) - Returns the estimated cardinality by HyperLogLog++ - with relativeSD, the maximum estimation error allowed. -""") + usage = """ +_FUNC_(expr[, relativeSD]) - Returns the estimated cardinality by HyperLogLog++. + `relativeSD` defines the maximum estimation error allowed. + """, + extended = """ +Arguments: + expr - an expression of any type that represents data to count. --- End diff -- ```sql spark-sql> SELECT approx_count_distinct(array(1)), approx_count_distinct(struct(1)), approx_count_distinct(map(1,1)); 1 1 1 ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85626188
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala
---
@@ -102,8 +102,17 @@ case class UserDefinedGenerator(
* }}}
*/
@ExpressionDescription(
- usage = "_FUNC_(n, v1, ..., vk) - Separate v1, ..., vk into n rows.",
- extended = "> SELECT _FUNC_(2, 1, 2, 3);\n [1,2]\n [3,null]")
+ usage = "_FUNC_(n, expr1, ..., exprk) - Separates `expr1`, ..., `exprk`
into `n` rows.",
+ extended = """
+Arguments:
+ n - an integer literal that represents the number of output rows.
+ expr - an expression of any type.
--- End diff --
```sql
spark-sql> SELECT stack(2, array(1));
[1]
NULL
spark-sql> SELECT stack(2, map(1, 1));
{1:1}
NULL
spark-sql> SELECT stack(2, struct(1));
{"col1":1}
NULL
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r85626782 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala --- @@ -114,7 +118,11 @@ case class Not(child: Expression) * Evaluates to `true` if `list` contains `value`. */ @ExpressionDescription( - usage = "expr _FUNC_(val1, val2, ...) - Returns true if expr equals to any valN.") + usage = "expr1 _FUNC_(expr2, expr3, ...) - Returns true if `expr` equals to any valN.", + extended = """ +Arguments: + expr - an expression of any type. --- End diff -- ```sql spark-sql> SELECT array(1) in(array(1)), struct(1) in(struct(1)), map(1,1) in(map(1, 1)); truetruefalse ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85626234
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala
---
@@ -631,7 +682,11 @@ case class CurrentDatabase() extends LeafExpression
with Unevaluable {
* TODO: Support Decimal and date related types
*/
@ExpressionDescription(
- usage = "_FUNC_(a1, a2, ...) - Returns a hash value of the arguments.")
+ usage = "_FUNC_(expr1, expr2, ...) - Returns a hash value of the
arguments.",
+ extended = """
+Arguments:
+ expr - an expression of any type.
--- End diff --
```sql
spark-sql> SELECT hash(struct(1));
-559580957
Time taken: 0.027 seconds, Fetched 1 row(s)
spark-sql> SELECT hash(array(1));
-559580957
Time taken: 0.021 seconds, Fetched 1 row(s)
spark-sql> SELECT hash(map(1, 1));
245521047
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85625713
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Max.scala
---
@@ -23,7 +23,11 @@ import org.apache.spark.sql.catalyst.util.TypeUtils
import org.apache.spark.sql.types._
@ExpressionDescription(
- usage = "_FUNC_(expr) - Returns the maximum value of expr.")
+ usage = "_FUNC_(expr) - Returns the maximum value of `expr`.",
+ extended = """
+Arguments:
+ expr - an expression of any type.
+ """)
--- End diff --
```sql
SELECT max(array(1)), max(struct(1));
[1] {"col1":1}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85626603
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala
---
@@ -126,7 +155,17 @@ case class NullIf(left: Expression, right: Expression)
extends RuntimeReplaceabl
}
-@ExpressionDescription(usage = "_FUNC_(a,b) - Returns b if a is null, or a
otherwise.")
+@ExpressionDescription(
+ usage = "_FUNC_(expr1, expr2) - Returns `expr2` if `expr1` is null, or
`expr1` otherwise.",
+ extended = """
+Arguments:
+ expr1 - an expression of any type.
+ expr2 - an expression of any type.
--- End diff --
```sql
spark-sql> SELECT nvl(array(1), array(1)), nvl(struct(1), struct(1)),
nvl(map(1,1), map(1, 1));
[1] {"col1":1} {1:1}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85625648
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
---
@@ -234,7 +259,16 @@ case class CreateStruct(children: Seq[Expression])
extends Expression {
*/
// scalastyle:off line.size.limit
@ExpressionDescription(
- usage = "_FUNC_(name1, val1, name2, val2, ...) - Creates a struct with
the given field names and values.")
+ usage = "_FUNC_(name1, val1, name2, val2, ...) - Creates a struct with
the given field names and values.",
+ extended = """
+Arguments:
+ name - a string expression literal that represents the field name.
+ val - an expression of any type.
--- End diff --
```sql
spark-sql> SELECT named_struct('1', array(1)), named_struct('1',
struct(1)), named_struct('1', map(1, 1));
{"1":[1]} {"1":{"col1":1}}{"1":{1:1}}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85625439
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
---
@@ -28,7 +28,15 @@ import org.apache.spark.unsafe.types.UTF8String
* Returns an Array containing the evaluation of all children expressions.
*/
@ExpressionDescription(
- usage = "_FUNC_(n0, ...) - Returns an array with the given elements.")
+ usage = "_FUNC_(expr, ...) - Returns an array with the given elements.",
+ extended = """
+Arguments:
+ expr - an expression of any type.
--- End diff --
```sql
spark-sql> SELECT array(array(1)), array(struct(1)), array(map(1, 1));
[[1]] [{"col1":1}][{1:1}]
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85624910
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/First.scala
---
@@ -29,10 +29,16 @@ import org.apache.spark.sql.types._
* a single partition, and we use a single reducer to do the aggregation.).
*/
@ExpressionDescription(
- usage = """_FUNC_(expr) - Returns the first value of `child` for a group
of rows.
-_FUNC_(expr,isIgnoreNull=false) - Returns the first value of `child`
for a group of rows.
- If isIgnoreNull is true, returns only non-null values.
-""")
+ usage = """
+_FUNC_(expr[, isIgnoreNull]) - Returns the first value of `expr` for a
group of rows.
+ If `isIgnoreNull` is true, returns only non-null values.
+ """,
+ extended = """
+Arguments:
+ expr - an expression of any type that represents data to collect the
first.
--- End diff --
```sql
spark-sql> SELECT first(array(1)), first(struct(1)), first(map(1,1));
[1] {"col1":1} {1:1}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85625801
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
---
@@ -531,7 +615,15 @@ case class Least(children: Seq[Expression]) extends
Expression {
* It takes at least 2 parameters, and returns null iff all parameters are
null.
*/
@ExpressionDescription(
- usage = "_FUNC_(n1, ...) - Returns the greatest value of all parameters,
skipping null values.")
+ usage = "_FUNC_(expr, ...) - Returns the greatest value of all
parameters, skipping null values.",
+ extended = """
+Arguments:
+ expr - an expression of any type.
+
--- End diff --
```sql
spark-sql> SELECT greatest(array(1), array(1)), least(struct(1), struct(1));
[1] {"col1":1}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85626037
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala
---
@@ -162,7 +172,15 @@ abstract class CaseWhenBase(
*/
// scalastyle:off line.size.limit
@ExpressionDescription(
- usage = "CASE WHEN a THEN b [WHEN c THEN d]* [ELSE e] END - When a =
true, returns b; when c = true, return d; else return e.")
+ usage = "CASE WHEN expr1 THEN expr2 [WHEN expr3 THEN expr4]* [ELSE
expr5] END - When `expr1` = true, returns `expr2`; when `expr3` = true, return
`expr4`; else return `expr5`.",
+ extended = """
+Arguments:
+ expr1 - a boolean expression.
+ expr2 - an expression of any type.
+ expr3 - a boolean expression.
+ expr4 - an expression of any type.
+ expr5 - an expression of any type.
--- End diff --
```sql
spark-sql> SELECT CASE WHEN true THEN array(1) WHEN true THEN array(1) ELSE
array(1) END;
[1]
spark-sql> SELECT CASE WHEN true THEN struct(1) WHEN true THEN struct(1)
ELSE struct(1) END;
{"col1":1}
spark-sql> SELECT CASE WHEN true THEN map(1, 1) WHEN true THEN map(1, 1)
ELSE map(1, 1) END;
{1:1}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85626369
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala
---
@@ -88,7 +97,17 @@ case class Coalesce(children: Seq[Expression]) extends
Expression {
}
-@ExpressionDescription(usage = "_FUNC_(a,b) - Returns b if a is null, or a
otherwise.")
+@ExpressionDescription(
+ usage = "_FUNC_(expr1, expr2) - Returns `expr2` if `expr1` is null, or
`expr1` otherwise.",
+ extended = """
+Arguments:
+ expr1 - an expression of any type.
+ expr2 - an expression of any type.
--- End diff --
```sql
spark-sql> SELECT ifnull(array(1), array(1)), ifnull(struct(1), struct(1)),
ifnull(map(1,1), map(1, 1));
[1] {"col1":1} {1:1}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85625732
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Min.scala
---
@@ -23,7 +23,11 @@ import org.apache.spark.sql.catalyst.util.TypeUtils
import org.apache.spark.sql.types._
@ExpressionDescription(
- usage = "_FUNC_(expr) - Returns the minimum value of expr.")
+ usage = "_FUNC_(expr) - Returns the minimum value of `expr`.",
+ extended = """
+Arguments:
+ expr - an expression of any type.
+ """)
--- End diff --
```sql
spark-sql> SELECT min(array(1)), min(struct(1));
[1] {"col1":1}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85626278
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala
---
@@ -34,9 +34,18 @@ import org.apache.spark.sql.types._
* coalesce(null, null, null) => null
* }}}
*/
+// scalastyle:off line.size.limit
@ExpressionDescription(
- usage = "_FUNC_(a1, a2, ...) - Returns the first non-null argument if
exists. Otherwise, NULL.",
- extended = "> SELECT _FUNC_(NULL, 1, NULL);\n 1")
+ usage = "_FUNC_(expr1, expr2, ...) - Returns the first non-null argument
if exists. Otherwise, null.",
+ extended = """
+Arguments:
+ expr - an expression of any type.
--- End diff --
```sql
spark-sql> SELECT coalesce(array(1)), coalesce(struct(1)), coalesce(map(1,
1));
[1] {"col1":1} {1:1}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85626558
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala
---
@@ -490,7 +521,15 @@ abstract class InterpretedHashFunction {
* and bucketing have same data distribution.
*/
@ExpressionDescription(
- usage = "_FUNC_(a1, a2, ...) - Returns a hash value of the arguments.")
+ usage = "_FUNC_(expr1, expr2, ...) - Returns a hash value of the
arguments.",
+ extended = """
+Arguments:
+ expr - an expression of any type.
--- End diff --
```sql
spark-sql> SELECT hash(struct(1));
-559580957
Time taken: 0.027 seconds, Fetched 1 row(s)
spark-sql> SELECT hash(array(1));
-559580957
Time taken: 0.021 seconds, Fetched 1 row(s)
spark-sql> SELECT hash(map(1, 1));
245521047
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r85624830 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Count.scala --- @@ -23,9 +23,17 @@ import org.apache.spark.sql.types._ // scalastyle:off line.size.limit @ExpressionDescription( - usage = """_FUNC_(*) - Returns the total number of retrieved rows, including rows containing NULL values. -_FUNC_(expr) - Returns the number of rows for which the supplied expression is non-NULL. -_FUNC_(DISTINCT expr[, expr...]) - Returns the number of rows for which the supplied expression(s) are unique and non-NULL.""") + usage = """ +_FUNC_(*) - Returns the total number of retrieved rows, including rows containing null. + +_FUNC_(expr) - Returns the number of rows for which the supplied expression is non-null. + +_FUNC_(DISTINCT expr[, expr...]) - Returns the number of rows for which the supplied expression(s) are unique and non-null. + """, + extended = """ +Arguments: + expr - an expression of any type that represents data to count. --- End diff -- ```sql spark-sql> SELECT count(array(1)), count(struct(1)), count(map(1,1)); 1 1 1 ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85625546
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
---
@@ -175,7 +192,15 @@ case class CreateMap(children: Seq[Expression])
extends Expression {
* Returns a Row containing the evaluation of all children expressions.
*/
@ExpressionDescription(
- usage = "_FUNC_(col1, col2, col3, ...) - Creates a struct with the given
field values.")
+ usage = "_FUNC_(expr1, expr2, expr2 ...) - Creates a struct with the
given field values.",
+ extended = """
+Arguments:
+ expr - an expression of any type.
--- End diff --
```sql
spark-sql> SELECT struct(array(1)), struct(struct(1)), struct(map(1, 1));
{"col1":[1]}{"col1":{"col1":1}} {"col1":{1:1}}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85626365
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala
---
@@ -106,7 +125,17 @@ case class IfNull(left: Expression, right: Expression)
extends RuntimeReplaceabl
}
-@ExpressionDescription(usage = "_FUNC_(a,b) - Returns null if a equals to
b, or a otherwise.")
+@ExpressionDescription(
+ usage = "_FUNC_(expr1, expr2) - Returns null if `expr1` equals to
`expr2`, or `expr1` otherwise.",
+ extended = """
+ Arguments:
+ expr1 - an expression of any type.
+ expr2 - an expression of any type.
--- End diff --
```sql
spark-sql> SELECT nullif(array(1), array(1)), nullif(struct(1), struct(1)),
nullif(map(1,1), map(1, 1));
NULLNULL{1:1}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85628048
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala
---
@@ -372,22 +372,29 @@ abstract class OffsetWindowFunction
}
/**
- * The Lead function returns the value of 'x' at the 'offset'th row after
the current row in
+ * The Lead function returns the value of `input` at the `offset`th row
after the current row in
* the window. Offsets start at 0, which is the current row. The offset
must be constant
- * integer value. The default offset is 1. When the value of 'x' is null
at the 'offset'th row,
- * null is returned. If there is no such offset row, the default
expression is evaluated.
+ * integer value. The default offset is 1. When the value of `input` is
null at the `offset`th row,
+ * null is returned. If there is no such offset row, the `default`
expression is evaluated.
*
- * @param input expression to evaluate 'offset' rows after the current row.
+ * @param input expression to evaluate `offset` rows after the current row.
* @param offset rows to jump ahead in the partition.
* @param default to use when the offset is larger than the window. The
default value is null.
*/
-@ExpressionDescription(usage =
- """_FUNC_(input, offset, default) - LEAD returns the value of 'x' at the
'offset'th row
- after the current row in the window.
- The default value of 'offset' is 1 and the default value of 'default'
is null.
- If the value of 'x' at the 'offset'th row is null, null is returned.
- If there is no such offset row (e.g. when the offset is 1, the last
row of the window
- does not have any subsequent row), 'default' is returned.""")
+@ExpressionDescription(
+ usage = """
+_FUNC_(input[, offset[, default]]) - Returns the value of `input` at
the `offset`th row
+ after the current row in the window. The default value of `offset`
is 1 and the default
+ value of `default` is null. If the value of `input` at the
`offset`th row is null,
+ null is returned. If there is no such an offset row (e.g., when the
offset is 1, the last
+ row of the window does not have any subsequent row), `default` is
returned.
+ """,
+ extended = """
+Arguments:
+ input - an expression of any type.
+ offset - a numeric expression. Default is 1.
+ default - an expression of any type. Default is null.
+ """)
--- End diff --
```scala
{
val df = Seq((1, "1"), (2, "2"), (1, "1"), (2, "2")).toDF("key",
"value")
.selectExpr("array(value) as value", "key")
df.select(
lead("value",
1).over(Window.partitionBy($"key").orderBy($"value"))).show()
}
{
val df = Seq((1, "1"), (2, "2"), (1, "1"), (2, "2")).toDF("key",
"value")
.selectExpr("struct(value) as value", "key")
df.select(
lead("value",
1).over(Window.partitionBy($"key").orderBy($"value"))).show()
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r85626702 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala --- @@ -261,7 +330,15 @@ case class NaNvl(left: Expression, right: Expression) * An expression that is evaluated to true if the input is null. */ @ExpressionDescription( - usage = "_FUNC_(a) - Returns true if a is NULL and false otherwise.") + usage = "_FUNC_(expr) - Returns true if `expr` is null and false otherwise.", + extended = """ +Arguments: + expr - an expression of any type. --- End diff -- ```sql spark-sql> SELECT isnull(array(1)), isnull(struct(1)), isnull(map(1, 1)); falsefalse false ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85625901
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala
---
@@ -24,7 +24,17 @@ import org.apache.spark.sql.types._
// scalastyle:off line.size.limit
@ExpressionDescription(
- usage = "_FUNC_(expr1,expr2,expr3) - If expr1 is TRUE then IF() returns
expr2; otherwise it returns expr3.")
+ usage = "_FUNC_(expr1, expr2, expr3) - If `expr1` evaluates to true,
then returns `expr2`; otherwise it returns `expr3`.",
+ extended = """
+Arguments:
+ expr1 - a boolean expression.
+ expr2 - an expression of any type that represents the return value
when `expr1` evaluates to true.
+ expr3 - an expression of any type that represents the return value
when `expr1` evaluates to false.
--- End diff --
```sql
spark-sql> SELECT if(true, array(1), array(1)), if(false, struct(1),
struct(1)), if(true, map(1, 1), map(1, 1));
[1] {"col1":1} {1:1}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85624768
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
---
@@ -114,8 +114,16 @@ object Cast {
/** Cast the child expression to the target data type. */
@ExpressionDescription(
- usage = " - Cast value v to the target data type.",
- extended = "> SELECT _FUNC_('10' as int);\n 10")
+ usage = "_FUNC_(expr AS type) - Casts the value `expr` to the target
data type `type`.",
+ extended = """
+Arguments:
+ expr - an expression of any type.
--- End diff --
```sql
spark-sql> SELECT cast(array(1) as string), cast(struct(1) as string),
cast(map(1,1) as string);
[1] [1] keys: [1], values: [1]
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85625485
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
---
@@ -82,7 +90,16 @@ case class CreateArray(children: Seq[Expression])
extends Expression {
* The children are a flatted sequence of kv pairs, e.g. (key1, value1,
key2, value2, ...)
*/
@ExpressionDescription(
- usage = "_FUNC_(key0, value0, key1, value1...) - Creates a map with the
given key/value pairs.")
+ usage = "_FUNC_(key0, value0, key1, value1...) - Creates a map with the
given key/value pairs.",
+ extended = """
+Arguments:
+ key - an expression of any type.
+ value - an expression of any type.
--- End diff --
```sql
spark-sql> SELECT map(1, array(1)), map(1, struct(1)), map(1, map(1, 1));
{1:[1]} {1:{"col1":1}} {1:{1:1}}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85625180
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Last.scala
---
@@ -29,7 +29,16 @@ import org.apache.spark.sql.types._
* a single partition, and we use a single reducer to do the aggregation.).
*/
@ExpressionDescription(
- usage = "_FUNC_(expr,isIgnoreNull) - Returns the last value of `child`
for a group of rows.")
+ usage = """
+_FUNC_(expr[, isIgnoreNull]) - Returns the last value of `expr` for a
group of rows.
+ If `isIgnoreNull` is true, returns only non-null values.
+ """,
+ extended = """
+Arguments:
+ expr - an expression of any type that represents data to collect the
last.
--- End diff --
```sql
spark-sql> SELECT last(array(1)), last(struct(1)), last(map(1,1));
[1] {"col1":1} {1:1}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85626717
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala
---
@@ -282,7 +359,15 @@ case class IsNull(child: Expression) extends
UnaryExpression with Predicate {
* An expression that is evaluated to true if the input is not null.
*/
@ExpressionDescription(
- usage = "_FUNC_(a) - Returns true if a is not NULL and false otherwise.")
+ usage = "_FUNC_(expr) - Returns true if `expr` is not null and false
otherwise.",
+ extended = """
+Arguments:
+ expr - an expression of any type.
--- End diff --
```sql
spark-sql> SELECT isnotnull(array(1)), isnotnull(struct(1)),
isnotnull(map(1, 1));
truetruetrue
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85626871
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala
---
@@ -409,7 +427,12 @@ object Equality {
}
@ExpressionDescription(
- usage = "a _FUNC_ b - Returns TRUE if a equals b and false otherwise.")
+ usage = "expr1 _FUNC_ expr2 - Returns true if `expr1` equals `expr2` and
false otherwise.",
+ extended = """
+Arguments:
+ expr1 - an expression of any type.
+ expr2 - an expression of any type.
--- End diff --
```sql
spark-sql> SELECT array(1) = array(1), struct(1) = struct(1), map(1, 1) =
map(1, 1);
truetruefalse
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85625786
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
---
@@ -471,7 +547,15 @@ case class Pmod(left: Expression, right: Expression)
extends BinaryArithmetic wi
* It takes at least 2 parameters, and returns null iff all parameters are
null.
*/
@ExpressionDescription(
- usage = "_FUNC_(n1, ...) - Returns the least value of all parameters,
skipping null values.")
+ usage = "_FUNC_(expr, ...) - Returns the least value of all parameters,
skipping null values.",
+ extended = """
+Arguments:
+ expr - an expression of any type.
+
--- End diff --
```sql
spark-sql> SELECT least(array(1), array(1)), least(struct(1), struct(1));
[1] {"col1":1}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r85626890 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala --- @@ -435,8 +458,15 @@ case class EqualTo(left: Expression, right: Expression) } @ExpressionDescription( - usage = """a _FUNC_ b - Returns same result with EQUAL(=) operator for non-null operands, -but returns TRUE if both are NULL, FALSE if one of the them is NULL.""") + usage = """ +expr1 _FUNC_ expr2 - Returns same result as the EQUAL(=) operator for non-null operands, + but returns true if both are null, false if one of the them is null. + """, + extended = """ +Arguments: + expr1 - an expression of any type. + expr2 - an expression of any type. --- End diff -- ```sql spark-sql> SELECT array(1) <=> array(1), struct(1) <=> struct(1), map(1, 1) <=> map(1, 1); truetruefalse ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85625172
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/collect.scala
---
@@ -86,7 +86,11 @@ abstract class Collect extends ImperativeAggregate {
* Collect a list of elements.
*/
@ExpressionDescription(
- usage = "_FUNC_(expr) - Collects and returns a list of non-unique
elements.")
+ usage = "_FUNC_(expr) - Collects and returns a list of non-unique
elements.",
+ extended = """
+Arguments:
+ expr - an expression of any type that represents data to collect as
a list.
--- End diff --
```sql
spark-sql> SELECT collect_list(array(1)), collect_list(struct(1)),
collect_list(map(1, 1));
[[1]] [{"col1":1}][{1:1}]
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85625758
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/collect.scala
---
@@ -106,10 +110,14 @@ case class CollectList(
}
/**
- * Collect a list of unique elements.
+ * Collect a set of unique elements.
*/
@ExpressionDescription(
- usage = "_FUNC_(expr) - Collects and returns a set of unique elements.")
+ usage = "_FUNC_(expr) - Collects and returns a set of unique elements.",
+ extended = """
+Arguments:
+ expr - an expression of any type that represents data to collect as
a set.
+ """)
--- End diff --
```sql
spark-sql> SELECT collect_set(array(1)), collect_set(struct(1));
[[1]] [{"col1":1}]
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85626660
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala
---
@@ -144,7 +183,20 @@ case class Nvl(left: Expression, right: Expression)
extends RuntimeReplaceable {
}
-@ExpressionDescription(usage = "_FUNC_(a,b,c) - Returns b if a is not
null, or c otherwise.")
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = "_FUNC_(expr1, expr2, expr3) - Returns `expr2` if `expr1` is not
null, or `expr3` otherwise.",
+ extended = """
+Arguments:
+ expr1 - an expression of any type.
+ expr2 - an expression of any type.
+ expr3 - an expression of any type.
--- End diff --
```sql
spark-sql> SELECT nvl2(array(1), array(1), array(1)), nvl2(struct(1),
struct(1), struct(1)), nvl2(map(1,1), map(1, 1), map(1, 1));
[1] {"col1":1} {1:1}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85628075
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala
---
@@ -401,22 +408,29 @@ case class Lead(input: Expression, offset:
Expression, default: Expression)
}
/**
- * The Lag function returns the value of 'x' at the 'offset'th row before
the current row in
+ * The Lag function returns the value of `input` at the `offset`th row
before the current row in
* the window. Offsets start at 0, which is the current row. The offset
must be constant
- * integer value. The default offset is 1. When the value of 'x' is null
at the 'offset'th row,
- * null is returned. If there is no such offset row, the default
expression is evaluated.
+ * integer value. The default offset is 1. When the value of `input` is
null at the `offset`th row,
+ * null is returned. If there is no such offset row, the `default`
expression is evaluated.
*
- * @param input expression to evaluate 'offset' rows before the current
row.
+ * @param input expression to evaluate `offset` rows before the current
row.
* @param offset rows to jump back in the partition.
* @param default to use when the offset row does not exist.
*/
-@ExpressionDescription(usage =
- """_FUNC_(input, offset, default) - LAG returns the value of 'x' at the
'offset'th row
- before the current row in the window.
- The default value of 'offset' is 1 and the default value of 'default'
is null.
- If the value of 'x' at the 'offset'th row is null, null is returned.
- If there is no such offset row (e.g. when the offset is 1, the first
row of the window
- does not have any previous row), 'default' is returned.""")
+@ExpressionDescription(
+ usage = """
+_FUNC_(input[, offset[, default]]) - Returns the value of `input` at
the `offset`th row
+ before the current row in the window. The default value of `offset`
is 1 and the default
+ value of `default` is null. If the value of `input` at the
`offset`th row is null,
+ null is returned. If there is no such offset row (e.g., when the
offset is 1, the first
+ row of the window does not have any previous row), `default` is
returned.
+ """,
+ extended = """
+Arguments:
+ input - an expression of any type.
+ offset - a numeric expression. Default is 1.
+ default - an expression of any type. Default is null.
--- End diff --
```scala
{
val df = Seq((1, "1"), (2, "2"), (1, "1"), (2, "2")).toDF("key",
"value")
.selectExpr("array(value) as value", "key")
df.select(
lag("value",
1).over(Window.partitionBy($"key").orderBy($"value"))).show()
}
{
val df = Seq((1, "1"), (2, "2"), (1, "1"), (2, "2")).toDF("key",
"value")
.selectExpr("struct(value) as value", "key")
df.select(
lag("value",
1).over(Window.partitionBy($"key").orderBy($"value"))).show()
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85627086
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -851,8 +993,16 @@ case class ParseUrl(children: Seq[Expression])
*/
// scalastyle:off line.size.limit
@ExpressionDescription(
- usage = "_FUNC_(String format, Obj... args) - Returns a formatted string
from printf-style format strings.",
- extended = "> SELECT _FUNC_(\"Hello World %d %s\", 100, \"days\");\n
'Hello World 100 days'")
+ usage = "_FUNC_(strfmt, obj, ...) - Returns a formatted string from
printf-style format strings.",
+ extended = """
+Arguments:
+ strfmt - a string expression.
+ obj - an expression of any type.
--- End diff --
```sql
spark-sql> SELECT format_string("Hello World %d %s", 100, array(1),
struct(1), map(1, 1));
Hello World 100 [1]
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r85246919 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala --- @@ -251,7 +259,12 @@ case class InSet(child: Expression, hset: Set[Any]) extends UnaryExpression with } @ExpressionDescription( - usage = "a _FUNC_ b - Logical AND.") + usage = "expr1 _FUNC_ expr2 - Logical AND.", + extended = """ +Arguments: + expr1 - an expression of any type. + expr2 - an expression of any type. --- End diff -- I should change. It was my mistake. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r85221940 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala --- @@ -251,7 +259,12 @@ case class InSet(child: Expression, hset: Set[Any]) extends UnaryExpression with } @ExpressionDescription( - usage = "a _FUNC_ b - Logical AND.") + usage = "expr1 _FUNC_ expr2 - Logical AND.", + extended = """ +Arguments: + expr1 - an expression of any type. + expr2 - an expression of any type. --- End diff -- Are you sure it can support any type? For logical operations (AND, OR or others), I think the only acceptable types are boolean --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r85217860 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala --- @@ -435,8 +458,15 @@ case class EqualTo(left: Expression, right: Expression) } @ExpressionDescription( - usage = """a _FUNC_ b - Returns same result with EQUAL(=) operator for non-null operands, -but returns TRUE if both are NULL, FALSE if one of the them is NULL.""") + usage = """ +expr1 _FUNC_ expr2 - Returns same result as the EQUAL(=) operator for non-null operands. --- End diff -- `.` -> `,` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85217483
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/randomExpressions.scala
---
@@ -55,8 +55,20 @@ abstract class RDG extends LeafExpression with
Nondeterministic {
}
/** Generate a random column with i.i.d. uniformly distributed values in
[0, 1). */
+// scalastyle:off line.size.limit
@ExpressionDescription(
- usage = "_FUNC_(a) - Returns a random column with i.i.d. uniformly
distributed values in [0, 1).")
+ usage = "_FUNC_([seed]) - Returns a random column with i.i.d. uniformly
distributed values in [0, 1].",
--- End diff --
`[0, 1]`. This is a semantic change, right? Any reason?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85214549
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/randomExpressions.scala
---
@@ -77,9 +89,21 @@ case class Rand(seed: Long) extends RDG {
}
}
-/** Generate a random column with i.i.d. gaussian random distribution. */
+/** Generate a random column with i.i.d. values drawn from the standard
normal distribution. */
+// scalastyle:off line.size.limit
@ExpressionDescription(
- usage = "_FUNC_(a) - Returns a random column with i.i.d. gaussian random
distribution.")
+ usage = "_FUNC_([seed]) - Returns a random column with i.i.d. values
drawn from the standard normal distribution.",
--- End diff --
`i.i.d. values` -> `independent and identically distributed (i.i.d.)
values`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85209471
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -657,10 +777,22 @@ case class StringLPad(str: Expression, len:
Expression, pad: Expression)
* Returns str, right-padded with pad to a length of len.
*/
@ExpressionDescription(
- usage = """_FUNC_(str, len, pad) - Returns str, right-padded with pad to
a length of len.
-If str is longer than len, the return value is shortened to len
characters.""",
- extended = "> SELECT _FUNC_('hi', 5, '??');\n 'hi???'\n" +
-"> SELECT _FUNC_('hi', 1, '??');\n 'h'")
+ usage = """
+_FUNC_(str, len, pad) - Returns str, right-padded with pad to a length
of len.
+ If str is longer than len, the return value is shortened to len
characters.
+ """,
+ extended = """
+Arguments:
+ str - a string expression.
+ len - a numeric expression that defines the number of pad.
--- End diff --
`pad` -> `pads`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85184948
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xml/xpath.scala
---
@@ -68,11 +76,21 @@ case class XPathBoolean(xml: Expression, path:
Expression) extends XPathExtract
}
}
+// scalastyle:off line.size.limit
@ExpressionDescription(
- usage = "_FUNC_(xml, xpath) - Returns a short value that matches the
xpath expression",
- extended = "> SELECT _FUNC_('12','sum(a/b)');\n3")
+ usage = "_FUNC_(xml, xpath) - Returns a short value, or the value zero
if no match is found, or a match is found but the value is non-numeric.",
--- End diff --
`a short value` -> `a short integer value`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85184825
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xml/xpath.scala
---
@@ -120,9 +168,19 @@ case class XPathFloat(xml: Expression, path:
Expression) extends XPathExtract {
}
}
+// scalastyle:off line.size.limit
@ExpressionDescription(
- usage = "_FUNC_(xml, xpath) - Returns a double value that matches the
xpath expression",
- extended = "> SELECT _FUNC_('12','sum(a/b)');\n3.0")
+ usage = "_FUNC_(xml, xpath) - Returns a double value, the value zero if
no match is found, or NaN if a match is found but the value is non-numeric.",
--- End diff --
`the value` -> `or the value`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85184742
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xml/xpath.scala
---
@@ -107,9 +145,19 @@ case class XPathLong(xml: Expression, path:
Expression) extends XPathExtract {
}
}
+// scalastyle:off line.size.limit
@ExpressionDescription(
- usage = "_FUNC_(xml, xpath) - Returns a float value that matches the
xpath expression",
- extended = "> SELECT _FUNC_('12','sum(a/b)');\n3.0")
+ usage = "_FUNC_(xml, xpath) - Returns a float value, the value zero if
no match is found, or NaN if a match is found but the value is non-numeric.",
--- End diff --
`the value` -> `or the value`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85184264
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xml/xpath.scala
---
@@ -94,9 +122,19 @@ case class XPathInt(xml: Expression, path: Expression)
extends XPathExtract {
}
}
+// scalastyle:off line.size.limit
@ExpressionDescription(
- usage = "_FUNC_(xml, xpath) - Returns a long value that matches the
xpath expression",
- extended = "> SELECT _FUNC_('12','sum(a/b)');\n3")
+ usage = "_FUNC_(xml, xpath) - Returns a long value, or the value zero if
no match is found, or a match is found but the value is non-numeric.",
--- End diff --
`a long value` is confusing. We should use `a long integer value`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85183882
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xml/xpath.scala
---
@@ -56,8 +56,16 @@ abstract class XPathExtract extends BinaryExpression
with ExpectsInputTypes with
}
@ExpressionDescription(
- usage = "_FUNC_(xml, xpath) - Evaluates a boolean xpath expression.",
- extended = "> SELECT _FUNC_('1','a/b');\ntrue")
+ usage = "_FUNC_(xml, xpath) - Evaluates a boolean an XPath expression.",
--- End diff --
This is wrong, right? In Hive, the document is like:
`Returns true if the XPath expression evaluates to true, or if a matching
node is found.`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r85111739 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala --- @@ -25,7 +25,11 @@ import org.apache.spark.sql.types._ import org.apache.spark.unsafe.types.CalendarInterval @ExpressionDescription( - usage = "_FUNC_(a) - Returns -a.") + usage = "_FUNC_(expr) - Returns the negated value of expr.", + extended = """ +Arguments: + expr - a numeric or interval expression. --- End diff -- Added. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r85061116 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala --- @@ -410,13 +418,21 @@ case class Lead(input: Expression, offset: Expression, default: Expression) * @param offset rows to jump back in the partition. * @param default to use when the offset row does not exist. */ -@ExpressionDescription(usage = - """_FUNC_(input, offset, default) - LAG returns the value of 'x' at the 'offset'th row - before the current row in the window. - The default value of 'offset' is 1 and the default value of 'default' is null. - If the value of 'x' at the 'offset'th row is null, null is returned. - If there is no such offset row (e.g. when the offset is 1, the first row of the window - does not have any previous row), 'default' is returned.""") +@ExpressionDescription( + usage = """ +_FUNC_(input[, offset[, default]]) - Returns the value of 'x' at the 'offset'th row + before the current row in the window. + The default value of 'offset' is 1 and the default value of 'default' is null. + If the value of 'x' at the 'offset'th row is null, null is returned. + If there is no such offset row (e.g. when the offset is 1, the first row of the window + does not have any previous row), 'default' is returned. + """, + extended = """ +Arguments: + input - an expression of any type. + offset - a numeric expression. Default is 1. + default - an expression of any type. Defualt is NULL. --- End diff -- Be consistent with the function description. `NULL` -> `null` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85060624
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala
---
@@ -664,10 +690,12 @@ case class Rank(children: Seq[Expression]) extends
RankLike {
* change in rank. This is an internal parameter and will
be assigned by the
* Analyser.
*/
-@ExpressionDescription(usage =
- """_FUNC_() - The DENSE_RANK() function computes the rank of a value in
a group of
- values. The result is one plus the previously assigned rank value.
Unlike Rank,
- DenseRank will not produce gaps in the ranking sequence.""")
+@ExpressionDescription(
+ usage = """
+_FUNC_() - Computes the rank of a value in a group of
+ values. The result is one plus the previously assigned rank value.
Unlike Rank,
--- End diff --
`Unlike Rank` -> `Unlike the function rank`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85060476
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala
---
@@ -664,10 +690,12 @@ case class Rank(children: Seq[Expression]) extends
RankLike {
* change in rank. This is an internal parameter and will
be assigned by the
* Analyser.
*/
-@ExpressionDescription(usage =
- """_FUNC_() - The DENSE_RANK() function computes the rank of a value in
a group of
- values. The result is one plus the previously assigned rank value.
Unlike Rank,
- DenseRank will not produce gaps in the ranking sequence.""")
+@ExpressionDescription(
+ usage = """
+_FUNC_() - Computes the rank of a value in a group of
+ values. The result is one plus the previously assigned rank value.
Unlike Rank,
+ DenseRank will not produce gaps in the ranking sequence.
--- End diff --
`DenseRank` -> `dense_rank`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r85060090
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xml/xpath.scala
---
@@ -150,8 +206,16 @@ case class XPathString(xml: Expression, path:
Expression) extends XPathExtract {
// scalastyle:off line.size.limit
@ExpressionDescription(
- usage = "_FUNC_(xml, xpath) - Returns a string array of values within
xml nodes that match the xpath expression",
- extended = "> SELECT
_FUNC_('b1b2b3c1c2','a/b/text()');\n['b1','b2','b3']")
+ usage = "_FUNC_(xml, xpath) - Returns a string array of values within
the nodes of xml that match the XPath expression.",
--- End diff --
`the XPath expression xpath`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r84867463 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala --- @@ -25,7 +25,11 @@ import org.apache.spark.sql.types._ import org.apache.spark.unsafe.types.CalendarInterval @ExpressionDescription( - usage = "_FUNC_(a) - Returns -a.") + usage = "_FUNC_(expr) - Returns the negated value of expr.", --- End diff -- Yes, negative is not correct here, it's negated --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r84867405
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xml/xpath.scala
---
@@ -108,8 +140,16 @@ case class XPathLong(xml: Expression, path:
Expression) extends XPathExtract {
}
@ExpressionDescription(
- usage = "_FUNC_(xml, xpath) - Returns a float value that matches the
xpath expression",
- extended = "> SELECT _FUNC_('12','sum(a/b)');\n3.0")
+ usage = "_FUNC_(xml, xpath) - Returns a float value that matches the
XPath expression.",
+ extended = """
+Arguments:
+ xml - a string expression that represents XML document.
--- End diff --
Likewise "an XML document"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r84867371
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xml/xpath.scala
---
@@ -95,8 +119,16 @@ case class XPathInt(xml: Expression, path: Expression)
extends XPathExtract {
}
@ExpressionDescription(
- usage = "_FUNC_(xml, xpath) - Returns a long value that matches the
xpath expression",
- extended = "> SELECT _FUNC_('12','sum(a/b)');\n3")
+ usage = "_FUNC_(xml, xpath) - Returns a long value that matches the
XPath expression.",
--- End diff --
yes, the name of the standard is "XPath" not "xpath"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r84831284
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xml/xpath.scala
---
@@ -95,8 +119,16 @@ case class XPathInt(xml: Expression, path: Expression)
extends XPathExtract {
}
@ExpressionDescription(
- usage = "_FUNC_(xml, xpath) - Returns a long value that matches the
xpath expression",
- extended = "> SELECT _FUNC_('12','sum(a/b)');\n3")
+ usage = "_FUNC_(xml, xpath) - Returns a long value that matches the
XPath expression.",
--- End diff --
This one returns `long`. So, it is accurate but maybe not sufficient.
```scala
scala> spark.sql("SELECT xpath_long('12',
'a')").printSchema()
root
|-- xpath_long(12, a): long (nullable = true)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r84831067
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xml/xpath.scala
---
@@ -95,8 +119,16 @@ case class XPathInt(xml: Expression, path: Expression)
extends XPathExtract {
}
@ExpressionDescription(
- usage = "_FUNC_(xml, xpath) - Returns a long value that matches the
xpath expression",
- extended = "> SELECT _FUNC_('12','sum(a/b)');\n3")
+ usage = "_FUNC_(xml, xpath) - Returns a long value that matches the
XPath expression.",
--- End diff --
`XPath` -> `xpath` please see
https://github.com/apache/spark/pull/15513#discussion_r83644370
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r84830885
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
---
@@ -105,7 +113,15 @@ case class DateAdd(startDate: Expression, days:
Expression)
*/
@ExpressionDescription(
usage = "_FUNC_(start_date, num_days) - Returns the date that is
num_days before start_date.",
- extended = "> SELECT _FUNC_('2016-07-30', 1);\n '2016-07-29'")
+ extended = """
+Arguments:
+ start_date - a date expression.
+ num_days - a numeric expression that represents the number of days
to subtract.
--- End diff --
The same one with
https://github.com/apache/spark/pull/15513/files/c55ecb60c79694c79a7c60c0c4a759bff6c71fc6#r84830856
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r84828020 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala --- @@ -25,7 +25,11 @@ import org.apache.spark.sql.types._ import org.apache.spark.unsafe.types.CalendarInterval @ExpressionDescription( - usage = "_FUNC_(a) - Returns -a.") + usage = "_FUNC_(expr) - Returns the negated value of expr.", --- End diff -- Please see https://github.com/apache/spark/pull/15513#discussion_r84573696 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r84824488
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
---
@@ -105,7 +113,15 @@ case class DateAdd(startDate: Expression, days:
Expression)
*/
@ExpressionDescription(
usage = "_FUNC_(start_date, num_days) - Returns the date that is
num_days before start_date.",
- extended = "> SELECT _FUNC_('2016-07-30', 1);\n '2016-07-29'")
+ extended = """
+Arguments:
+ start_date - a date expression.
+ num_days - a numeric expression that represents the number of days
to subtract.
--- End diff --
`an integral constant expression`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r84824307
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
---
@@ -76,7 +76,15 @@ case class CurrentTimestamp() extends LeafExpression
with CodegenFallback {
*/
@ExpressionDescription(
usage = "_FUNC_(start_date, num_days) - Returns the date that is
num_days after start_date.",
- extended = "> SELECT _FUNC_('2016-07-30', 1);\n '2016-07-31'")
+ extended = """
+Arguments:
+ start_date - a date expression.
+ num_days - a numeric expression that represents the number of days
to add.
--- End diff --
We only support integer here, right? -> `an integral expression`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r84824076 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala --- @@ -24,7 +24,17 @@ import org.apache.spark.sql.types._ // scalastyle:off line.size.limit @ExpressionDescription( - usage = "_FUNC_(expr1,expr2,expr3) - If expr1 is TRUE then IF() returns expr2; otherwise it returns expr3.") + usage = "_FUNC_(expr1, expr2, expr3) - If expr1 is TRUE then IF() returns expr2; otherwise it returns expr3.", + extended = """ +Arguments: + expr1 - a boolean expression. + expr2 - an expression of any type that represents the return value when expr1 is TRUE. --- End diff -- `when expr1 is TRUE` -> `when expr1 evaluates to true.` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r84823563 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala --- @@ -367,8 +428,18 @@ case class Remainder(left: Expression, right: Expression) } @ExpressionDescription( - usage = "_FUNC_(a, b) - Returns the positive modulo", - extended = "> SELECT _FUNC_(10,3);\n 1") + usage = "_FUNC_(expr1, expr2) - Returns the positive modulo.", --- End diff -- Since you explicitly shows the expression in the above function, here we can do the same thing. `Returns the positive value of expr1 mod expr2.` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r84823116 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala --- @@ -25,7 +25,11 @@ import org.apache.spark.sql.types._ import org.apache.spark.unsafe.types.CalendarInterval @ExpressionDescription( - usage = "_FUNC_(a) - Returns -a.") + usage = "_FUNC_(expr) - Returns the negated value of expr.", + extended = """ +Arguments: + expr - a numeric or interval expression. --- End diff -- Example? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r84822854 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala --- @@ -25,7 +25,11 @@ import org.apache.spark.sql.types._ import org.apache.spark.unsafe.types.CalendarInterval @ExpressionDescription( - usage = "_FUNC_(a) - Returns -a.") + usage = "_FUNC_(expr) - Returns the negated value of expr.", --- End diff -- `negated ` -> `nagative` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r84821645 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala --- @@ -410,13 +418,21 @@ case class Lead(input: Expression, offset: Expression, default: Expression) * @param offset rows to jump back in the partition. * @param default to use when the offset row does not exist. */ -@ExpressionDescription(usage = - """_FUNC_(input, offset, default) - LAG returns the value of 'x' at the 'offset'th row - before the current row in the window. - The default value of 'offset' is 1 and the default value of 'default' is null. - If the value of 'x' at the 'offset'th row is null, null is returned. - If there is no such offset row (e.g. when the offset is 1, the first row of the window - does not have any previous row), 'default' is returned.""") +@ExpressionDescription( + usage = """ +_FUNC_(input[, offset[, default]]) - Returns the value of 'x' at the 'offset'th row + before the current row in the window. + The default value of 'offset' is 1 and the default value of 'default' is null. + If the value of 'x' at the 'offset'th row is null, null is returned. + If there is no such offset row (e.g. when the offset is 1, the first row of the window + does not have any previous row), 'default' is returned. + """, + extended = """ +Arguments: + input - an expression of any type. + offset - a numeric expression. Default is 1. + default - an expression of any type. Defualt is NULL. --- End diff -- typo: `Defualt` -> `Default` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r84821378
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala
---
@@ -664,10 +690,12 @@ case class Rank(children: Seq[Expression]) extends
RankLike {
* change in rank. This is an internal parameter and will
be assigned by the
* Analyser.
*/
-@ExpressionDescription(usage =
- """_FUNC_() - The DENSE_RANK() function computes the rank of a value in
a group of
- values. The result is one plus the previously assigned rank value.
Unlike Rank,
- DenseRank will not produce gaps in the ranking sequence.""")
+@ExpressionDescription(
+ usage = """
+_FUNC_() - Computes the rank of a value in a group of
--- End diff --
Nit: update the formats
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r84821301
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xml/xpath.scala
---
@@ -108,8 +140,16 @@ case class XPathLong(xml: Expression, path:
Expression) extends XPathExtract {
}
@ExpressionDescription(
- usage = "_FUNC_(xml, xpath) - Returns a float value that matches the
xpath expression",
- extended = "> SELECT _FUNC_('12','sum(a/b)');\n3.0")
+ usage = "_FUNC_(xml, xpath) - Returns a float value that matches the
XPath expression.",
+ extended = """
+Arguments:
+ xml - a string expression that represents XML document.
--- End diff --
`XML document` -> `a XML document`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r84821075
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xml/xpath.scala
---
@@ -108,8 +140,16 @@ case class XPathLong(xml: Expression, path:
Expression) extends XPathExtract {
}
@ExpressionDescription(
- usage = "_FUNC_(xml, xpath) - Returns a float value that matches the
xpath expression",
- extended = "> SELECT _FUNC_('12','sum(a/b)');\n3.0")
+ usage = "_FUNC_(xml, xpath) - Returns a float value that matches the
XPath expression.",
+ extended = """
+Arguments:
+ xml - a string expression that represents XML document.
+ xpath - a string literal that represents XPath expression.
--- End diff --
`XPath expression` -> `a XPath expression`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r84820966
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xml/xpath.scala
---
@@ -95,8 +119,16 @@ case class XPathInt(xml: Expression, path: Expression)
extends XPathExtract {
}
@ExpressionDescription(
- usage = "_FUNC_(xml, xpath) - Returns a long value that matches the
xpath expression",
- extended = "> SELECT _FUNC_('12','sum(a/b)');\n3")
+ usage = "_FUNC_(xml, xpath) - Returns a long value that matches the
XPath expression.",
--- End diff --
`XPath` -> `xpath`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r84820380 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala --- @@ -410,13 +418,21 @@ case class Lead(input: Expression, offset: Expression, default: Expression) * @param offset rows to jump back in the partition. * @param default to use when the offset row does not exist. */ -@ExpressionDescription(usage = - """_FUNC_(input, offset, default) - LAG returns the value of 'x' at the 'offset'th row - before the current row in the window. - The default value of 'offset' is 1 and the default value of 'default' is null. - If the value of 'x' at the 'offset'th row is null, null is returned. - If there is no such offset row (e.g. when the offset is 1, the first row of the window - does not have any previous row), 'default' is returned.""") +@ExpressionDescription( + usage = """ +_FUNC_(input[, offset[, default]]) - Returns the value of 'x' at the 'offset'th row --- End diff -- The same here. Should use `input` to replace `x`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r84744810 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Covariance.scala --- @@ -76,8 +76,14 @@ abstract class Covariance(x: Expression, y: Expression) extends DeclarativeAggre } } + --- End diff -- No need to add this extra code change. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r84615985 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/bitwiseExpressions.scala --- @@ -27,8 +27,16 @@ import org.apache.spark.sql.types._ * Code generation inherited from BinaryArithmetic. */ @ExpressionDescription( - usage = "a _FUNC_ b - Bitwise AND.", - extended = "> SELECT 3 _FUNC_ 5; 1") + usage = "expr1 _FUNC_ expr2 - Bitwise AND.", + extended = """ +Arguments: + expr1 - an integral numeric expression. --- End diff -- Oh, you meant what is that. I referred `IntegralType` class. maybe I should fix that just to `intergral expression`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r84615684
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/CallMethodViaReflection.scala
---
@@ -43,11 +43,20 @@ import org.apache.spark.util.Utils
* and the second element should be a literal string for
the method name,
* and the remaining are input arguments to the Java
method.
*/
-// scalastyle:off line.size.limit
@ExpressionDescription(
- usage = "_FUNC_(class,method[,arg1[,arg2..]]) calls method with
reflection",
- extended = "> SELECT _FUNC_('java.util.UUID', 'randomUUID');\n
c33fb387-8500-4bfa-81d2-6e0e3e930df2")
-// scalastyle:on line.size.limit
+ usage = "_FUNC_(class, method[, arg1[, arg2 ..]]) - Calls method with
reflection.",
+ extended = """
+Arguments:
+ class - a string literal that represents a fully-qualified class
name.
+ method - a string literal that represents a method name.
+ arg - a boolean, numeric or string expression that represents
arguments for the method.
--- End diff --
Ah, no, numeric types except decimal. I will note that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r84615167 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Covariance.scala --- @@ -76,8 +76,14 @@ abstract class Covariance(x: Expression, y: Expression) extends DeclarativeAggre } } + --- End diff -- This wouldn't be a nit because most of case class definitions across expressions seem have double-spaced indentation. Also, it seems fine - https://github.com/databricks/scala-style-guide#blank-lines-vertical-whitespace > Use one or two blank line(s) to separate class definitions. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r84613013
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/CallMethodViaReflection.scala
---
@@ -43,11 +43,20 @@ import org.apache.spark.util.Utils
* and the second element should be a literal string for
the method name,
* and the remaining are input arguments to the Java
method.
*/
-// scalastyle:off line.size.limit
@ExpressionDescription(
- usage = "_FUNC_(class,method[,arg1[,arg2..]]) calls method with
reflection",
- extended = "> SELECT _FUNC_('java.util.UUID', 'randomUUID');\n
c33fb387-8500-4bfa-81d2-6e0e3e930df2")
-// scalastyle:on line.size.limit
+ usage = "_FUNC_(class, method[, arg1[, arg2 ..]]) - Calls method with
reflection.",
+ extended = """
+Arguments:
+ class - a string literal that represents a fully-qualified class
name.
+ method - a string literal that represents a method name.
+ arg - a boolean, numeric or string expression that represents
arguments for the method.
--- End diff --
Do we support decimal?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r84604637 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala --- @@ -49,21 +49,29 @@ import org.apache.spark.sql.types._ * DEFAULT_PERCENTILE_ACCURACY. */ @ExpressionDescription( - usage = -""" - _FUNC_(col, percentage [, accuracy]) - Returns the approximate percentile value of numeric + usage = """ +_FUNC_(col, percentage [, accuracy]) - Returns the approximate percentile value of numeric column `col` at the given percentage. The value of percentage must be between 0.0 and 1.0. The `accuracy` parameter (default: 1) is a positive integer literal which controls approximation accuracy at the cost of memory. Higher value of `accuracy` yields better accuracy, `1.0/accuracy` is the relative error of the approximation. - - _FUNC_(col, array(percentage1 [, percentage2]...) [, accuracy]) - Returns the approximate - percentile array of column `col` at the given percentage array. Each value of the - percentage array must be between 0.0 and 1.0. The `accuracy` parameter (default: 1) is - a positive integer literal which controls approximation accuracy at the cost of memory. - Higher value of `accuracy` yields better accuracy, `1.0/accuracy` is the relative error of - the approximation. -""") + When percentage is an array, each value of the percentage array must be between 0.0 and 1.0. + In this case, returns the approximate percentile array of column `col` at the given + percentage array. + """, + extended = """ +Arguments: + col - a numeric expression. + percentage - a numeric literal or an array literal of numeric type that defines the +percentile between 0.0 and 1.0. For example, 0.5 means 50-percentile. + accuracy - a numeric literal that defines approximation accuracy. --- End diff -- expression takes both value and column where as literal takes value only in general. (It throws an exception when a column is given instead). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r84604598
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xml/xpath.scala
---
@@ -150,8 +206,16 @@ case class XPathString(xml: Expression, path:
Expression) extends XPathExtract {
// scalastyle:off line.size.limit
@ExpressionDescription(
- usage = "_FUNC_(xml, xpath) - Returns a string array of values within
xml nodes that match the xpath expression",
- extended = "> SELECT
_FUNC_('b1b2b3c1c2','a/b/text()');\n['b1','b2','b3']")
+ usage = "_FUNC_(xml, xpath) - Returns a string array of values within
the nodes of xml that match the XPath expression.",
+ extended = """
+Arguments:
+ xml - a string expression that represents XML document.
+ path - a string literal that represents XPath expression.
--- End diff --
expression takes both value and column where as literal takes value only in
general.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r84597837 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala --- @@ -49,21 +49,29 @@ import org.apache.spark.sql.types._ * DEFAULT_PERCENTILE_ACCURACY. */ @ExpressionDescription( - usage = -""" - _FUNC_(col, percentage [, accuracy]) - Returns the approximate percentile value of numeric + usage = """ +_FUNC_(col, percentage [, accuracy]) - Returns the approximate percentile value of numeric column `col` at the given percentage. The value of percentage must be between 0.0 and 1.0. The `accuracy` parameter (default: 1) is a positive integer literal which controls approximation accuracy at the cost of memory. Higher value of `accuracy` yields better accuracy, `1.0/accuracy` is the relative error of the approximation. - - _FUNC_(col, array(percentage1 [, percentage2]...) [, accuracy]) - Returns the approximate - percentile array of column `col` at the given percentage array. Each value of the - percentage array must be between 0.0 and 1.0. The `accuracy` parameter (default: 1) is - a positive integer literal which controls approximation accuracy at the cost of memory. - Higher value of `accuracy` yields better accuracy, `1.0/accuracy` is the relative error of - the approximation. -""") + When percentage is an array, each value of the percentage array must be between 0.0 and 1.0. + In this case, returns the approximate percentile array of column `col` at the given + percentage array. + """, + extended = """ +Arguments: + col - a numeric expression. + percentage - a numeric literal or an array literal of numeric type that defines the +percentile between 0.0 and 1.0. For example, 0.5 means 50-percentile. + accuracy - a numeric literal that defines approximation accuracy. --- End diff -- What is the difference between `a numeric expression` and `a numeric literal`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r84597661
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala
---
@@ -692,9 +722,11 @@ case class DenseRank(children: Seq[Expression])
extends RankLike {
* change in rank. This is an internal parameter and will
be assigned by the
* Analyser.
*/
-@ExpressionDescription(usage =
- """_FUNC_() - PERCENT_RANK() The PercentRank function computes the
percentage
- ranking of a value in a group of values.""")
+@ExpressionDescription(
+ usage = """
+_FUNC_() - PERCENT_RANK() The PercentRank function computes the
percentage
--- End diff --
`The PercentRank function computes the percentage` -> `Computes the
percentage`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r84597610
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xml/xpath.scala
---
@@ -150,8 +206,16 @@ case class XPathString(xml: Expression, path:
Expression) extends XPathExtract {
// scalastyle:off line.size.limit
@ExpressionDescription(
- usage = "_FUNC_(xml, xpath) - Returns a string array of values within
xml nodes that match the xpath expression",
- extended = "> SELECT
_FUNC_('b1b2b3c1c2','a/b/text()');\n['b1','b2','b3']")
+ usage = "_FUNC_(xml, xpath) - Returns a string array of values within
the nodes of xml that match the XPath expression.",
+ extended = """
+Arguments:
+ xml - a string expression that represents XML document.
+ path - a string literal that represents XPath expression.
--- End diff --
What is the differences between `a string expression` and `a string
literal`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r84597530 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala --- @@ -970,9 +1270,19 @@ case class Round(child: Expression, scale: Expression) * also known as Gaussian rounding or bankers' rounding. * round(2.5) = 2.0, round(3.5) = 4.0. */ +// scalastyle:off line.size.limit @ExpressionDescription( - usage = "_FUNC_(x, d) - Round x to d decimal places using HALF_EVEN rounding mode.", - extended = "> SELECT _FUNC_(2.5, 0);\n 2.0") + usage = "_FUNC_(expr, d) - Round expr to d decimal places using HALF_EVEN rounding mode.", --- End diff -- Add `Returns` at the beginning. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r84591475
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
---
@@ -1455,50 +1455,59 @@ class DDLSuite extends QueryTest with
SharedSQLContext with BeforeAndAfterEach {
sql("DESCRIBE FUNCTION log"),
Row("Class: org.apache.spark.sql.catalyst.expressions.Logarithm") ::
Row("Function: log") ::
-Row("Usage: log(b, x) - Returns the logarithm of x with base b.")
:: Nil
+Row("Usage: log(base, expr) - Returns the logarithm of expr with
base.") :: Nil
)
// predicate operator
checkAnswer(
sql("DESCRIBE FUNCTION or"),
Row("Class: org.apache.spark.sql.catalyst.expressions.Or") ::
Row("Function: or") ::
-Row("Usage: a or b - Logical OR.") :: Nil
+Row("Usage: expr1 or expr2 - Logical OR.") :: Nil
)
checkAnswer(
sql("DESCRIBE FUNCTION !"),
Row("Class: org.apache.spark.sql.catalyst.expressions.Not") ::
Row("Function: !") ::
-Row("Usage: ! a - Logical not") :: Nil
+Row("Usage: ! expr - Logical not.") :: Nil
)
// arithmetic operators
checkAnswer(
sql("DESCRIBE FUNCTION +"),
Row("Class: org.apache.spark.sql.catalyst.expressions.Add") ::
Row("Function: +") ::
-Row("Usage: a + b - Returns a+b.") :: Nil
+Row("Usage: expr1 + expr2 - Returns expr1+expr2.") :: Nil
)
// comparison operators
checkAnswer(
sql("DESCRIBE FUNCTION <"),
Row("Class: org.apache.spark.sql.catalyst.expressions.LessThan") ::
Row("Function: <") ::
-Row("Usage: a < b - Returns TRUE if a is less than b.") :: Nil
+Row("Usage: expr1 < expr2 - Returns TRUE if expr1 is less than
expr2.") :: Nil
)
// STRING
checkAnswer(
sql("DESCRIBE FUNCTION 'concat'"),
Row("Class: org.apache.spark.sql.catalyst.expressions.Concat") ::
Row("Function: concat") ::
Row("Usage: concat(str1, str2, ..., strN) " +
- "- Returns the concatenation of str1, str2, ..., strN") :: Nil
+ "- Returns the concatenation of str1, str2, ..., strN.") :: Nil
)
// extended mode
checkAnswer(
sql("DESCRIBE FUNCTION EXTENDED ^"),
Row("Class: org.apache.spark.sql.catalyst.expressions.BitwiseXor") ::
-Row("Extended Usage:\n> SELECT 3 ^ 5; 2") ::
+Row(
+ """Extended Usage:
+|Arguments:
+| expr1 - a integral numeric expression.
--- End diff --
I will sweep it the same instances!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user jodersky commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r84591482
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala
---
@@ -125,7 +129,7 @@ case class DescribeFunctionCommand(
if (isExtended) {
result :+
- Row(s"Extended
Usage:\n${replaceFunctionName(info.getExtended, info.getName)}")
+ Row(s"Extended Usage:${replaceFunctionName(info.getExtended,
info.getName)}")
--- End diff --
Indeed, annotations require constant parameters (probably due to JVM
requirements). Since `stripMargin` is a method on a string wrapper, it
unfortunately cannot be used as an annotation argument
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r84590685
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala
---
@@ -125,7 +129,7 @@ case class DescribeFunctionCommand(
if (isExtended) {
result :+
- Row(s"Extended
Usage:\n${replaceFunctionName(info.getExtended, info.getName)}")
+ Row(s"Extended Usage:${replaceFunctionName(info.getExtended,
info.getName)}")
--- End diff --
Yes, it seems `stripMargin` not working in annotations in 2.11 too. In the
comments, it is about doing `stripMargin` actually right before printing (not
trying to assign the string with `stripMargin`).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r84590609
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/CallMethodViaReflection.scala
---
@@ -43,11 +43,20 @@ import org.apache.spark.util.Utils
* and the second element should be a literal string for
the method name,
* and the remaining are input arguments to the Java
method.
*/
-// scalastyle:off line.size.limit
@ExpressionDescription(
- usage = "_FUNC_(class,method[,arg1[,arg2..]]) calls method with
reflection",
- extended = "> SELECT _FUNC_('java.util.UUID', 'randomUUID');\n
c33fb387-8500-4bfa-81d2-6e0e3e930df2")
-// scalastyle:on line.size.limit
+ usage = "_FUNC_(class, method[, arg1[, arg2 ..]]) - Calls method with
reflection.",
+ extended = """
+Arguments:
+ class - a string literal that represents a fully-qualified class
name.
+ method - a string literal that represents a method name.
+ arg - a string literal that represents arguments for the method.
--- End diff --
Oh, it seems `arg` is not. Let me try to fine such cases here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r84590562
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala
---
@@ -125,7 +129,7 @@ case class DescribeFunctionCommand(
if (isExtended) {
result :+
- Row(s"Extended
Usage:\n${replaceFunctionName(info.getExtended, info.getName)}")
+ Row(s"Extended Usage:${replaceFunctionName(info.getExtended,
info.getName)}")
--- End diff --
I don't think stripMargin works (at least in one version of the scala we
support perhaps 2.10) in annotations.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r84583708
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/CallMethodViaReflection.scala
---
@@ -43,11 +43,20 @@ import org.apache.spark.util.Utils
* and the second element should be a literal string for
the method name,
* and the remaining are input arguments to the Java
method.
*/
-// scalastyle:off line.size.limit
@ExpressionDescription(
- usage = "_FUNC_(class,method[,arg1[,arg2..]]) calls method with
reflection",
- extended = "> SELECT _FUNC_('java.util.UUID', 'randomUUID');\n
c33fb387-8500-4bfa-81d2-6e0e3e930df2")
-// scalastyle:on line.size.limit
+ usage = "_FUNC_(class, method[, arg1[, arg2 ..]]) - Calls method with
reflection.",
+ extended = """
+Arguments:
+ class - a string literal that represents a fully-qualified class
name.
+ method - a string literal that represents a method name.
+ arg - a string literal that represents arguments for the method.
--- End diff --
is it always literal?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r84582345
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala
---
@@ -125,7 +129,7 @@ case class DescribeFunctionCommand(
if (isExtended) {
result :+
- Row(s"Extended
Usage:\n${replaceFunctionName(info.getExtended, info.getName)}")
+ Row(s"Extended Usage:${replaceFunctionName(info.getExtended,
info.getName)}")
--- End diff --
@jodersky if you meant `stripMargin` here, I am able to do this but I
didn't do because I thought it's not great when other guys fix the
documentation in each function but if majority of you think it's better, I will
definitely do that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r84576139 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala --- @@ -367,8 +410,16 @@ case class Remainder(left: Expression, right: Expression) } @ExpressionDescription( - usage = "_FUNC_(a, b) - Returns the positive modulo", - extended = "> SELECT _FUNC_(10,3);\n 1") + usage = "_FUNC_(expr1, expr2) - Returns the positive modulo.", + extended = """ +Arguments: + expr1 - a numeric expression. + expr2 - a numeric expression. + +Examples: + > SELECT _FUNC_(10, 3); --- End diff -- (added in https://github.com/apache/spark/pull/15513/commits/5163a871929268865f321de8b861911ac7b0249c#diff-1516b10738479bbe190fb4e239258473R429) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r84574950 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala --- @@ -706,7 +842,16 @@ case class TimeAdd(start: Expression, interval: Expression) */ // scalastyle:off line.size.limit @ExpressionDescription( - usage = "_FUNC_(timestamp, string timezone) - Assumes given timestamp is UTC and converts to given timezone.") + usage = "_FUNC_(timestamp, timezone) - Assumes given timestamp is UTC and converts to given timezone.", --- End diff -- Thank you for the details about this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r84574163
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xml/xpath.scala
---
@@ -150,8 +220,18 @@ case class XPathString(xml: Expression, path:
Expression) extends XPathExtract {
// scalastyle:off line.size.limit
@ExpressionDescription(
- usage = "_FUNC_(xml, xpath) - Returns a string array of values within
xml nodes that match the xpath expression",
- extended = "> SELECT
_FUNC_('b1b2b3c1c2','a/b/text()');\n['b1','b2','b3']")
+ usage = "_FUNC_(xml, xpath) - Returns a string array of values within
xml nodes that match the XPath expression",
--- End diff --
"within the nodes of xml" ? to make it clear 'xml' is the argument and not
some general set of XML nodes
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r84574104 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala --- @@ -490,7 +525,12 @@ case class LessThan(left: Expression, right: Expression) } @ExpressionDescription( - usage = "a _FUNC_ b - Returns TRUE if a is not greater than b.") + usage = "expr1 _FUNC_ expr2 - Returns TRUE if expr1 is not greater than expr2.", --- End diff -- "not greater" seems a little indirect, compared to describing this as "less than or equal to" --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r84573971
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
---
@@ -384,71 +502,157 @@ case class Log2(child: Expression)
}
@ExpressionDescription(
- usage = "_FUNC_(x) - Returns the logarithm of x with base 10.",
- extended = "> SELECT _FUNC_(10);\n 1.0")
+ usage = "_FUNC_(expr) - Returns the logarithm of expr with base 10.",
+ extended = """
+Arguments:
+ expr - a numeric expression.
+
+Examples:
+ > SELECT _FUNC_(10);
+ 1.0
+ """)
case class Log10(child: Expression) extends UnaryLogExpression(math.log10,
"LOG10")
@ExpressionDescription(
- usage = "_FUNC_(x) - Returns log(1 + x).",
- extended = "> SELECT _FUNC_(0);\n 0.0")
+ usage = "_FUNC_(expr) - Returns log(1 + expr).",
+ extended = """
+Arguments:
+ expr - a numeric expression.
+
+Examples:
+ > SELECT _FUNC_(0);
+ 0.0
+ """)
case class Log1p(child: Expression) extends UnaryLogExpression(math.log1p,
"LOG1P") {
protected override val yAsymptote: Double = -1.0
}
+// scalastyle:off line.size.limit
@ExpressionDescription(
- usage = "_FUNC_(x, d) - Return the rounded x at d decimal places.",
- extended = "> SELECT _FUNC_(12.3456, 1);\n 12.3")
+ usage = "_FUNC_(expr) - Returns the double value that is closest in
value to the argument and is equal to a mathematical integer.",
+ extended = """
+Arguments:
+ expr - a numeric expression.
+
+Examples:
+ > SELECT _FUNC_(12.3456);
+ 12.0
+ """)
+// scalastyle:on line.size.limit
case class Rint(child: Expression) extends UnaryMathExpression(math.rint,
"ROUND") {
override def funcName: String = "rint"
}
@ExpressionDescription(
- usage = "_FUNC_(x) - Returns the sign of x.",
- extended = "> SELECT _FUNC_(40);\n 1.0")
+ usage = "_FUNC_(expr) - Returns the sign of expr.",
--- End diff --
You might mention that it returns -1, 0, 1 as expr is negative, 0 or
positive.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r84573731 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala --- @@ -286,7 +324,12 @@ case class Divide(left: Expression, right: Expression) } @ExpressionDescription( - usage = "a _FUNC_ b - Returns the remainder when dividing a by b.") + usage = "expr1 _FUNC_ expr2 - Returns the remainder when dividing expr1 by expr2.", + extended = """ +Arguments: + expr1 - a numeric expression. + expr2 - a numeric expression. + """) --- End diff -- Might be worth an example here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r84573927 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala --- @@ -159,28 +167,69 @@ case class Pi() extends LeafMathExpression(math.Pi, "PI") @ExpressionDescription( - usage = "_FUNC_(x) - Returns the arc cosine of x if -1<=x<=1 or NaN otherwise.", - extended = "> SELECT _FUNC_(1);\n 0.0\n> SELECT _FUNC_(2);\n NaN") + usage = "_FUNC_(expr) - Returns the arc cosine of expr if -1<=expr<=1 or NaN otherwise.", --- End diff -- More nits: all "arc" functions are named like "arccosine", not "arc cosine". Feel free to note in these docs that this just means "inverse cosine" and so on. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r84573878 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala --- @@ -847,7 +1010,16 @@ case class MonthsBetween(date1: Expression, date2: Expression) */ // scalastyle:off line.size.limit @ExpressionDescription( - usage = "_FUNC_(timestamp, string timezone) - Assumes given timestamp is in given timezone and converts to UTC.") + usage = "_FUNC_(timestamp, timezone) - Assumes given timestamp is in given timezone and converts to UTC.", --- End diff -- Likewise I think this is more accurately described as Given a timestamp, which corresponds to a certain time of day in the given timezone, returns another timestamp that corresponds to the same time of day in UTC. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r84574029 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala --- @@ -603,8 +823,16 @@ case class Unhex(child: Expression) extends UnaryExpression with ImplicitCastInp @ExpressionDescription( - usage = "_FUNC_(x,y) - Returns the arc tangent2.", - extended = "> SELECT _FUNC_(0, 0);\n 0.0") + usage = "_FUNC_(expr1, expr2) - Returns the arc tangent2.", --- End diff -- We should probably elaborate this a tiny bit, to say it's the arctangent of two arguments, because there's no "tangent2" function. Really it means (cribbing from wikipedia) the angle in radians between the positive x-axis of a plane and the point given by the coordinates (expr1, expr2) on it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r84574107 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala --- @@ -518,7 +563,12 @@ case class GreaterThan(left: Expression, right: Expression) } @ExpressionDescription( - usage = "a _FUNC_ b - Returns TRUE if a is not smaller than b.") + usage = "expr1 _FUNC_ expr2 - Returns TRUE if expr1 is not smaller than expr2.", --- End diff -- "not smaller" seems a little indirect, compared to describing this as "greater than or equal to" --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r84573696 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala --- @@ -25,7 +25,11 @@ import org.apache.spark.sql.types._ import org.apache.spark.unsafe.types.CalendarInterval @ExpressionDescription( - usage = "_FUNC_(a) - Returns -a.") + usage = "_FUNC_(expr) - Returns the negative value of expr.", --- End diff -- Nit: might say "Returns the negated value of expr" because this sort of sounds like it makes the value negative. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/15513#discussion_r84574135 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala --- @@ -986,7 +1155,14 @@ case class StringReverse(child: Expression) extends UnaryExpression with String2 */ @ExpressionDescription( usage = "_FUNC_(n) - Returns a n spaces string.", --- End diff -- "Returns a string consisting of n spaces" --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15513: [SPARK-17963][SQL][Documentation] Add examples (e...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/15513#discussion_r84574050
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala
---
@@ -159,8 +183,15 @@ case class Sha1(child: Expression) extends
UnaryExpression with ImplicitCastInpu
* For input of type [[BinaryType]]
*/
@ExpressionDescription(
- usage = "_FUNC_(input) - Returns a cyclic redundancy check value as a
bigint of the input",
- extended = "> SELECT _FUNC_('Spark');\n '1557323817'")
+ usage = "_FUNC_(expr) - Returns a cyclic redundancy check value as a
bigint of the expr.",
--- End diff --
Is this supposed to be worded as "Returns a cyclic redundancy check value
of the expr as a bigint"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
