This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.3 by this push:
new be6ce349e511 [SPARK-58359][SQL] Require the arguments field in
built-in function documentation
be6ce349e511 is described below
commit be6ce349e5114272873a12adef2fbcdca3b13847
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Tue Aug 4 20:14:18 2026 +0900
[SPARK-58359][SQL] Require the arguments field in built-in function
documentation
### What changes were proposed in this pull request?
This PR makes the `arguments` field of `ExpressionDescription` a required
part of built-in function documentation, alongside the already-required
`usage`, `examples`, `since`, and `group` fields.
- `ExpressionInfoSuite` (the `SPARK-32870` test) now asserts
`info.getArguments.nonEmpty` for every registered function, except functions
that take no arguments, which are enumerated in a new `noArgumentsSet` (e.g.
`pi`, `current_date`, `current_timestamp`, `input_file_name`, `uuid`,
`spark_partition_id`). The existing format assertions (the value must start
with `\n Arguments:\n` and end with `\n `) continue to apply.
- Backfills `arguments` documentation for 118 built-in expressions across
45 files that were previously missing it.
Note on scope: the enforced list is derived from what is actually
registered in the expression `FunctionRegistry`. Expressions whose
`ExpressionInfo` is produced through a builder object are validated via their
builder class; some of these already document `arguments` (e.g.
`minute`/`second`) and were not part of the backfill, while others whose
builders were missing it (e.g. `max_by`/`min_by`) were backfilled on the
builder class. Expressions that live only in the table function regi [...]
### Why are the changes needed?
Most built-in functions already document their `arguments`, but it was not
enforced, so newly added expressions could omit it and the generated function
docs would be inconsistent. Requiring the field keeps the built-in function
documentation complete and uniform, and turns a missing `Arguments:` section
into a test failure rather than a silent gap.
### Does this PR introduce _any_ user-facing change?
Yes. The generated documentation for the affected built-in functions now
includes an `Arguments:` section describing each argument. This is a
documentation-only change; there are no behavior changes.
### How was this patch tested?
Existing `ExpressionInfoSuite` covers this, extended to assert that
`arguments` is present for all registered functions (minus the zero-argument
exemptions).
```
build/sbt 'sql/testOnly
org.apache.spark.sql.expressions.ExpressionInfoSuite'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
This pull request and its description were written by Isaac.
Closes #57551 from HyukjinKwon/docs-require-arguments.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
(cherry picked from commit 5c7867479c66a16b7f5ac0e940262b9f7a990692)
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.../expressions/ApproxTopKExpressions.scala | 7 ++++
.../expressions/CallMethodViaReflection.scala | 7 ++++
.../spark/sql/catalyst/expressions/Cast.scala | 5 +++
.../spark/sql/catalyst/expressions/TryEval.scala | 6 +++
.../aggregate/ApproxTopKAggregates.scala | 9 +++++
.../aggregate/ApproximatePercentile.scala | 10 +++++
.../sql/catalyst/expressions/aggregate/Count.scala | 5 +++
.../expressions/aggregate/CountMinSketchAgg.scala | 7 ++++
.../expressions/aggregate/HistogramNumeric.scala | 6 +++
.../aggregate/HyperLogLogPlusPlus.scala | 6 +++
.../sql/catalyst/expressions/aggregate/Max.scala | 5 +++
.../catalyst/expressions/aggregate/MaxMinByK.scala | 14 +++++++
.../sql/catalyst/expressions/aggregate/Min.scala | 5 +++
.../catalyst/expressions/aggregate/collect.scala | 12 ++++++
.../aggregate/datasketchesAggregates.scala | 8 ++++
.../expressions/aggregate/kllAggregates.scala | 46 +++++++++++++++++++---
.../expressions/aggregate/percentiles.scala | 21 ++++++++++
.../aggregate/thetasketchesAggregates.scala | 11 ++++++
.../sql/catalyst/expressions/arithmetic.scala | 19 +++++++++
.../catalyst/expressions/avroSqlFunctions.scala | 18 +++++++++
.../catalyst/expressions/bitmapExpressions.scala | 15 +++++++
.../expressions/collectionOperations.scala | 42 ++++++++++++++++++++
.../catalyst/expressions/complexTypeCreator.scala | 27 ++++++++++++-
.../expressions/conditionalExpressions.scala | 6 +++
.../sql/catalyst/expressions/csvExpressions.scala | 21 ++++++++++
.../expressions/datasketchesExpressions.scala | 14 +++++++
.../catalyst/expressions/datetimeExpressions.scala | 4 ++
.../sql/catalyst/expressions/generators.scala | 21 ++++++++++
.../spark/sql/catalyst/expressions/grouping.scala | 11 ++++++
.../spark/sql/catalyst/expressions/hash.scala | 10 +++++
.../expressions/higherOrderFunctions.scala | 26 ++++++++++++
.../sql/catalyst/expressions/jsonExpressions.scala | 20 ++++++++++
.../spark/sql/catalyst/expressions/misc.scala | 4 ++
.../sql/catalyst/expressions/nullExpressions.scala | 28 +++++++++++++
.../catalyst/expressions/stringExpressions.scala | 6 +++
.../expressions/thetasketchesExpressions.scala | 29 ++++++++++++++
.../expressions/toFromProtobufSqlFunctions.scala | 23 +++++++++++
.../sql/catalyst/expressions/tupleDifference.scala | 22 +++++++++++
.../catalyst/expressions/tupleIntersection.scala | 34 ++++++++++++++++
.../catalyst/expressions/tupleSketchEstimate.scala | 20 ++++++++++
.../catalyst/expressions/tupleSketchSummary.scala | 14 +++++++
.../sql/catalyst/expressions/tupleUnion.scala | 36 +++++++++++++++++
.../expressions/variant/variantExpressions.scala | 35 ++++++++++++++++
.../catalyst/expressions/vectorExpressions.scala | 41 +++++++++++++++++++
.../sql/catalyst/expressions/xmlExpressions.scala | 17 ++++++++
.../sql/expressions/ExpressionInfoSuite.scala | 33 +++++++++++++++-
46 files changed, 778 insertions(+), 8 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ApproxTopKExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ApproxTopKExpressions.scala
index 8d4f0a18d570..fc647c860208 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ApproxTopKExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ApproxTopKExpressions.scala
@@ -42,6 +42,13 @@ import org.apache.spark.sql.types._
_FUNC_(state, k) - Returns top k items with their frequency.
`k` An optional INTEGER literal greater than 0. If k is not specified,
it defaults to 5.
""",
+ arguments = """
+ Arguments:
+ * state - The sketch state produced by `approx_top_k_accumulate` or
+ `approx_top_k_combine`.
+ * k - Optional. A constant INTEGER literal greater than 0 giving the
number
+ of top items to return. If omitted, it defaults to 5.
+ """,
examples = """
Examples:
> SELECT _FUNC_(approx_top_k_accumulate(expr)) FROM VALUES (0), (0),
(1), (1), (2), (3), (4), (4) AS tab(expr);
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/CallMethodViaReflection.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/CallMethodViaReflection.scala
index 90b9f19cf8d0..841648912df8 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/CallMethodViaReflection.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/CallMethodViaReflection.scala
@@ -51,6 +51,13 @@ import org.apache.spark.util.Utils
*/
@ExpressionDescription(
usage = "_FUNC_(class, method[, arg1[, arg2 ..]]) - Calls a method with
reflection.",
+ arguments = """
+ Arguments:
+ * class - A literal string with the fully qualified name of the class.
+ * method - A literal string with the name of the static method to call.
+ * argN - Optional arguments passed to the method. Only primitive and
string
+ types are supported, and each argument is matched to the method
signature.
+ """,
examples = """
Examples:
> SELECT _FUNC_('java.util.UUID', 'randomUUID');
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
index 72979889a3ba..b1fcee354b83 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
@@ -660,6 +660,11 @@ object Cast extends QueryErrorsBase {
@ExpressionDescription(
usage = "_FUNC_(expr AS type) - Casts the value `expr` to the target data
type `type`." +
" `expr` :: `type` alternative casting syntax is also supported.",
+ arguments = """
+ Arguments:
+ * expr - An expression whose value is converted to the target data type.
+ * type - The target data type to cast the value to.
+ """,
examples = """
Examples:
> SELECT _FUNC_('10' as int);
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TryEval.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TryEval.scala
index 289b102c1f6f..b41f690f8690 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TryEval.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TryEval.scala
@@ -329,6 +329,12 @@ case class TryToBinary(
@ExpressionDescription(
usage = "_FUNC_(class, method[, arg1[, arg2 ..]]) - This is a special
version of `reflect` that" +
" performs the same operation, but returns a NULL value instead of raising
an error if the invoke method thrown exception.",
+ arguments = """
+ Arguments:
+ * class - A string literal with the fully qualified name of the class.
+ * method - A string literal with the name of the static method to invoke.
+ * arg1, arg2, ... - Optional arguments passed to the invoked method.
+ """,
examples = """
Examples:
> SELECT _FUNC_('java.util.UUID', 'randomUUID');
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproxTopKAggregates.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproxTopKAggregates.scala
index f9023b683e11..8ed6590793da 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproxTopKAggregates.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproxTopKAggregates.scala
@@ -798,6 +798,15 @@ object CombineInternal {
_FUNC_(state, maxItemsTracked) - Combines multiple sketches into a single
sketch.
`maxItemsTracked` An optional positive INTEGER literal with upper limit
of 1000000. If maxItemsTracked is specified, it will be set for the combined
sketch. If maxItemsTracked is not specified, the input sketches must have the
same maxItemsTracked value, otherwise an error will be thrown. The output
sketch will use the same value from the input sketches.
""",
+ arguments = """
+ Arguments:
+ * state - The sketch state to combine, as produced by
approx_top_k_accumulate.
+ An expression that evaluates to the sketch state struct.
+ * maxItemsTracked - Optional. The maximum number of items to track in
the combined
+ sketch, with an upper limit of 1000000. An expression that evaluates
to an integer.
+ Must be a constant. If not specified, the input sketches must share
the same
+ maxItemsTracked value, which is used for the output sketch.
+ """,
examples = """
Examples:
> SELECT approx_top_k_estimate(_FUNC_(sketch, 10000), 5) FROM (SELECT
approx_top_k_accumulate(expr) AS sketch FROM VALUES (0), (0), (1), (1) AS
tab(expr) UNION ALL SELECT approx_top_k_accumulate(expr) AS sketch FROM VALUES
(2), (3), (4), (4) AS tab(expr));
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
index 19e30c53b436..b9caae5467ce 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
@@ -66,6 +66,16 @@ import org.apache.spark.util.ArrayImplicits._
In this case, returns the approximate percentile array of column `col`
at the given
percentage array.
""",
+ arguments = """
+ Arguments:
+ * col - The numeric, ANSI interval or TIME column whose percentile is
+ computed.
+ * percentage - A value (or array of values) between 0.0 and 1.0
specifying
+ the percentile(s) to compute.
+ * accuracy - Optional. A positive numeric literal (default: 10000) that
+ controls approximation accuracy at the cost of memory. Higher values
+ yield better accuracy; `1.0/accuracy` is the relative error.
+ """,
examples = """
Examples:
> SELECT _FUNC_(col, array(0.5, 0.4, 0.1), 100) FROM VALUES (0), (1),
(2), (10) AS tab(col);
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Count.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Count.scala
index 758ef22f0a2c..f5f248c48387 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Count.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Count.scala
@@ -34,6 +34,11 @@ import org.apache.spark.sql.types._
_FUNC_(DISTINCT expr[, expr...]) - Returns the number of rows for which
the supplied expression(s) are unique and non-null.
""",
+ arguments = """
+ Arguments:
+ * expr - One or more expressions. A row is counted only when all supplied
+ expressions are non-null. Use `*` to count all rows, including rows
with nulls.
+ """,
examples = """
Examples:
> SELECT _FUNC_(*) FROM VALUES (NULL), (5), (5), (20) AS tab(col);
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/CountMinSketchAgg.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/CountMinSketchAgg.scala
index f0a27677628d..a1eb024488bd 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/CountMinSketchAgg.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/CountMinSketchAgg.scala
@@ -206,6 +206,13 @@ case class CountMinSketchAgg(
`CountMinSketch` before usage. Count-min sketch is a probabilistic data
structure used for
cardinality estimation using sub-linear space.
""",
+ arguments = """
+ Arguments:
+ * col - The column to build the count-min sketch from.
+ * eps - A double literal for the relative error of the sketch.
+ * confidence - A double literal for the confidence of the sketch.
+ * seed - An integer literal used as the random seed.
+ """,
examples = """
Examples:
> SELECT hex(_FUNC_(col, 0.5d, 0.5d, 1)) FROM VALUES (1), (2), (1) AS
tab(col);
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/HistogramNumeric.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/HistogramNumeric.scala
index dadf7ac53c59..fe89a535f264 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/HistogramNumeric.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/HistogramNumeric.scala
@@ -51,6 +51,12 @@ import org.apache.spark.sql.util.NumericHistogram
statistical computing packages. Note: the output type of the 'x' field
in the return value is
propagated from the input value consumed in the aggregate function.
""",
+ arguments = """
+ Arguments:
+ * expr - A numeric, date, timestamp, or interval expression whose values
are aggregated
+ into the histogram.
+ * nb - A foldable integer expression (at least 2) giving the number of
histogram bins.
+ """,
examples = """
Examples:
> SELECT _FUNC_(col, 5) FROM VALUES (0), (1), (2), (10) AS tab(col);
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/HyperLogLogPlusPlus.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/HyperLogLogPlusPlus.scala
index f304b43358ad..56f203aa9c4e 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/HyperLogLogPlusPlus.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/HyperLogLogPlusPlus.scala
@@ -50,6 +50,12 @@ import org.apache.spark.sql.types._
usage = """
_FUNC_(expr[, relativeSD]) - Returns the estimated cardinality by
HyperLogLog++.
`relativeSD` defines the maximum relative standard deviation allowed.""",
+ arguments = """
+ Arguments:
+ * expr - An expression of any type whose distinct values are counted.
+ * relativeSD - An optional double literal for the maximum relative
standard
+ deviation allowed. Defaults to 0.05.
+ """,
examples = """
Examples:
> SELECT _FUNC_(col1) FROM VALUES (1), (1), (2), (2), (3) tab(col1);
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Max.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Max.scala
index 902f53309de4..bd8e3d3e1c8e 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Max.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Max.scala
@@ -26,6 +26,11 @@ import org.apache.spark.sql.types._
@ExpressionDescription(
usage = "_FUNC_(expr) - Returns the maximum value of `expr`.",
+ arguments = """
+ Arguments:
+ * expr - An expression of any orderable type whose maximum value across
the group is
+ returned. NULL values are ignored.
+ """,
examples = """
Examples:
> SELECT _FUNC_(col) FROM VALUES (10), (50), (20) AS tab(col);
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/MaxMinByK.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/MaxMinByK.scala
index a0a7acd93097..567beb9a7484 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/MaxMinByK.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/MaxMinByK.scala
@@ -240,6 +240,13 @@ case class MaxMinByK(
maximum values of `y`, sorted in descending order by `y`.
Returns NULL if there are no non-NULL ordering values.
""",
+ arguments = """
+ Arguments:
+ * x - The value expression to return.
+ * y - The ordering expression whose maximum selects the value of `x`.
+ * k - An optional positive integer. When present, returns an array of
the `k` values
+ of `x` associated with the largest values of `y`.
+ """,
examples = """
Examples:
> SELECT _FUNC_(x, y) FROM VALUES ('a', 10), ('b', 50), ('c', 20) AS
tab(x, y);
@@ -275,6 +282,13 @@ object MaxByBuilder extends ExpressionBuilder {
minimum values of `y`, sorted in ascending order by `y`.
Returns NULL if there are no non-NULL ordering values.
""",
+ arguments = """
+ Arguments:
+ * x - The value expression to return.
+ * y - The ordering expression whose minimum selects the value of `x`.
+ * k - An optional positive integer. When present, returns an array of
the `k` values
+ of `x` associated with the smallest values of `y`.
+ """,
examples = """
Examples:
> SELECT _FUNC_(x, y) FROM VALUES ('a', 10), ('b', 50), ('c', 20) AS
tab(x, y);
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Min.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Min.scala
index 7a9588808dbd..31fcd6f4f110 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Min.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Min.scala
@@ -26,6 +26,11 @@ import org.apache.spark.sql.types._
@ExpressionDescription(
usage = "_FUNC_(expr) - Returns the minimum value of `expr`.",
+ arguments = """
+ Arguments:
+ * expr - An expression of any orderable type whose minimum value across
the group is
+ returned. NULL values are ignored.
+ """,
examples = """
Examples:
> SELECT _FUNC_(col) FROM VALUES (10), (-1), (20) AS tab(col);
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/collect.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/collect.scala
index c3385bc39026..0ed76e0bfa7d 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/collect.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/collect.scala
@@ -104,6 +104,10 @@ abstract class Collect[T <: Growable[Any] with
Iterable[Any]] extends TypedImper
*/
@ExpressionDescription(
usage = "_FUNC_(expr) - Collects and returns a list of non-unique elements.",
+ arguments = """
+ Arguments:
+ * expr - An expression of any type whose values are collected into a
list.
+ """,
examples = """
Examples:
> SELECT _FUNC_(col) FROM VALUES (1), (2), (1) AS tab(col);
@@ -181,6 +185,10 @@ case class CollectList(
*/
@ExpressionDescription(
usage = "_FUNC_(expr) - Collects and returns a set of unique elements.",
+ arguments = """
+ Arguments:
+ * expr - An expression of any type whose values are collected into a set.
+ """,
examples = """
Examples:
> SELECT _FUNC_(col) FROM VALUES (1), (2), (1) AS tab(col);
@@ -336,6 +344,10 @@ case class CollectSet(
@ExpressionDescription(
usage =
"_FUNC_(expr) - Collects and returns the distinct union of the elements of
array `expr`.",
+ arguments = """
+ Arguments:
+ * expr - An array expression whose elements are collected into a set
across rows.
+ """,
examples = """
Examples:
> SELECT _FUNC_(col) FROM VALUES (array(1, 2)), (array(2, 3)),
(array(1)) AS tab(col);
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/datasketchesAggregates.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/datasketchesAggregates.scala
index 952b331b7227..4470c415f5c9 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/datasketchesAggregates.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/datasketchesAggregates.scala
@@ -243,6 +243,14 @@ object HllSketchAgg {
_FUNC_(expr, allowDifferentLgConfigK) - Returns the merged HllSketch's
updatable binary representation.
`allowDifferentLgConfigK` (optional) Allow sketches with different
lgConfigK values
to be unioned (defaults to false).""",
+ arguments = """
+ Arguments:
+ * expr - The binary representation of an HllSketch to merge.
+ An expression that evaluates to binary.
+ * allowDifferentLgConfigK - Optional. Whether to allow sketches with
different
+ lgConfigK values to be unioned. An expression that evaluates to a
boolean.
+ Defaults to false.
+ """,
examples = """
Examples:
> SELECT hll_sketch_estimate(_FUNC_(sketch, true)) FROM (SELECT
hll_sketch_agg(col) as sketch FROM VALUES (1) tab(col) UNION ALL SELECT
hll_sketch_agg(col, 20) as sketch FROM VALUES (1) tab(col));
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/kllAggregates.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/kllAggregates.scala
index a811736c5ee2..f13bad8291c6 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/kllAggregates.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/kllAggregates.scala
@@ -57,9 +57,10 @@ import org.apache.spark.sql.types.{AbstractDataType,
BinaryType, ByteType, DataT
arguments = """
Arguments:
* expr - The expression to aggregate into the KLL sketch.
- An expression that evaluates to an integral.
- * k - The parameter controlling the size and accuracy of the sketch.
- An expression that evaluates to an integer. Must be a constant.
+ An expression that evaluates to an integral.
+ * k - Optional. The parameter controlling the size and accuracy of the
sketch.
+ An expression that evaluates to an integer between 8 and 65535. Must
be a
+ constant. Defaults to 200.
""",
examples = """
Examples:
@@ -208,6 +209,14 @@ case class KllSketchAggBigint(
The optional k parameter controls the size and accuracy of the sketch
(default 200, range 8-65535).
Larger k values provide more accurate quantile estimates but result in
larger, slower sketches.
""",
+ arguments = """
+ Arguments:
+ * expr - The expression to aggregate into the KLL sketch.
+ An expression that evaluates to a float.
+ * k - Optional. The parameter controlling the size and accuracy of the
sketch.
+ An expression that evaluates to an integer between 8 and 65535. Must
be a
+ constant. Defaults to 200.
+ """,
examples = """
Examples:
> SELECT LENGTH(kll_sketch_to_string_float(_FUNC_(col))) > 0 FROM VALUES
(CAST(1.0 AS FLOAT)), (CAST(2.0 AS FLOAT)), (CAST(3.0 AS FLOAT)), (CAST(4.0 AS
FLOAT)), (CAST(5.0 AS FLOAT)) tab(col);
@@ -347,9 +356,10 @@ case class KllSketchAggFloat(
arguments = """
Arguments:
* expr - The expression to aggregate into the KLL sketch.
- An expression that evaluates to a float or double.
- * k - The parameter controlling the size and accuracy of the sketch.
- An expression that evaluates to an integer. Must be a constant.
+ An expression that evaluates to a float or double.
+ * k - Optional. The parameter controlling the size and accuracy of the
sketch.
+ An expression that evaluates to an integer between 8 and 65535. Must
be a
+ constant. Defaults to 200.
""",
examples = """
Examples:
@@ -493,6 +503,14 @@ case class KllSketchAggDouble(
The optional k parameter controls the size and accuracy of the merged
sketch (range 8-65535).
If k is not specified, the merged sketch adopts the k value from the
first input sketch.
""",
+ arguments = """
+ Arguments:
+ * expr - The expression to merge into the KLL sketch.
+ An expression that evaluates to a binary KLL sketch representation.
+ * k - Optional. The parameter controlling the size and accuracy of the
merged
+ sketch. An expression that evaluates to an integer between 8 and
65535. Must be
+ a constant. Defaults to the k value of the first input sketch.
+ """,
examples = """
Examples:
> SELECT kll_sketch_get_n_bigint(_FUNC_(sketch)) FROM (SELECT
kll_sketch_agg_bigint(col) as sketch FROM VALUES (1), (2), (3) tab(col) UNION
ALL SELECT kll_sketch_agg_bigint(col) as sketch FROM VALUES (4), (5), (6)
tab(col)) t;
@@ -566,6 +584,14 @@ case class KllMergeAggBigint(
The optional k parameter controls the size and accuracy of the merged
sketch (range 8-65535).
If k is not specified, the merged sketch adopts the k value from the
first input sketch.
""",
+ arguments = """
+ Arguments:
+ * expr - The expression to merge into the KLL sketch.
+ An expression that evaluates to a binary KLL sketch representation.
+ * k - Optional. The parameter controlling the size and accuracy of the
merged
+ sketch. An expression that evaluates to an integer between 8 and
65535. Must be
+ a constant. Defaults to the k value of the first input sketch.
+ """,
examples = """
Examples:
> SELECT kll_sketch_get_n_float(_FUNC_(sketch)) FROM (SELECT
kll_sketch_agg_float(col) as sketch FROM VALUES (CAST(1.0 AS FLOAT)), (CAST(2.0
AS FLOAT)), (CAST(3.0 AS FLOAT)) tab(col) UNION ALL SELECT
kll_sketch_agg_float(col) as sketch FROM VALUES (CAST(4.0 AS FLOAT)), (CAST(5.0
AS FLOAT)), (CAST(6.0 AS FLOAT)) tab(col)) t;
@@ -639,6 +665,14 @@ case class KllMergeAggFloat(
The optional k parameter controls the size and accuracy of the merged
sketch (range 8-65535).
If k is not specified, the merged sketch adopts the k value from the
first input sketch.
""",
+ arguments = """
+ Arguments:
+ * expr - The expression to merge into the KLL sketch.
+ An expression that evaluates to a binary KLL sketch representation.
+ * k - Optional. The parameter controlling the size and accuracy of the
merged
+ sketch. An expression that evaluates to an integer between 8 and
65535. Must be
+ a constant. Defaults to the k value of the first input sketch.
+ """,
examples = """
Examples:
> SELECT kll_sketch_get_n_double(_FUNC_(sketch)) FROM (SELECT
kll_sketch_agg_double(col) as sketch FROM VALUES (CAST(1.0 AS DOUBLE)),
(CAST(2.0 AS DOUBLE)), (CAST(3.0 AS DOUBLE)) tab(col) UNION ALL SELECT
kll_sketch_agg_double(col) as sketch FROM VALUES (CAST(4.0 AS DOUBLE)),
(CAST(5.0 AS DOUBLE)), (CAST(6.0 AS DOUBLE)) tab(col)) t;
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/percentiles.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/percentiles.scala
index cff1c6b9750f..4427dc25097e 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/percentiles.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/percentiles.scala
@@ -271,6 +271,15 @@ abstract class PercentileBase
positive integral
""",
+ arguments = """
+ Arguments:
+ * col - The column to compute the percentile of.
+ An expression that evaluates to a numeric, ANSI interval, or time.
+ * percentage - The percentile(s) to compute, each between 0.0 and 1.0.
Either a
+ single numeric value or an array of numeric values. Must be foldable.
+ * frequency - Optional. The number of times each value should be counted.
+ An expression that evaluates to a positive integral value. Defaults
to 1.
+ """,
examples = """
Examples:
> SELECT _FUNC_(col, 0.3) FROM VALUES (0), (10) AS tab(col);
@@ -493,6 +502,12 @@ case class PercentileDisc(
usage = "_FUNC_(percentage) WITHIN GROUP (ORDER BY col) - Return a
percentile value based on " +
"a continuous distribution of numeric, ANSI interval or TIME column `col`
at the given " +
"`percentage` (specified in ORDER BY clause).",
+ arguments = """
+ Arguments:
+ * percentage - The percentile to compute, between 0.0 and 1.0. Must be
foldable.
+ * col - The column to compute the percentile of, specified in the ORDER
BY clause.
+ An expression that evaluates to a numeric, ANSI interval, or time.
+ """,
examples = """
Examples:
> SELECT _FUNC_(0.25) WITHIN GROUP (ORDER BY col) FROM VALUES (0), (10)
AS tab(col);
@@ -519,6 +534,12 @@ object PercentileContBuilder extends ExpressionBuilder {
usage = "_FUNC_(percentage) WITHIN GROUP (ORDER BY col) - Return a
percentile value based on " +
"a discrete distribution of numeric, ANSI interval or TIME column `col` at
the given " +
"`percentage` (specified in ORDER BY clause).",
+ arguments = """
+ Arguments:
+ * percentage - The percentile to compute, between 0.0 and 1.0. Must be
foldable.
+ * col - The column to compute the percentile of, specified in the ORDER
BY clause.
+ An expression that evaluates to a numeric, ANSI interval, or time.
+ """,
examples = """
Examples:
> SELECT _FUNC_(0.25) WITHIN GROUP (ORDER BY col) FROM VALUES (0), (10)
AS tab(col);
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/thetasketchesAggregates.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/thetasketchesAggregates.scala
index d8cd93129ec8..a5326ae76c9f 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/thetasketchesAggregates.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/thetasketchesAggregates.scala
@@ -323,6 +323,13 @@ case class ThetaSketchAgg(
_FUNC_(expr, lgNomEntries) - Returns the ThetaSketch's Compact binary
representation.
`lgNomEntries` (optional) the log-base-2 of Nominal Entries, with
Nominal Entries deciding
the number buckets or slots for the ThetaSketch.""",
+ arguments = """
+ Arguments:
+ * expr - A binary expression of Compact ThetaSketch representations to
union.
+ * lgNomEntries - An optional integer expression, the log-base-2 of
nominal entries which
+ sets the number of buckets for the resulting ThetaSketch. When
omitted, it defaults to
+ the default log nominal entries.
+ """,
examples = """
Examples:
> SELECT theta_sketch_estimate(_FUNC_(sketch)) FROM (SELECT
theta_sketch_agg(col) as sketch FROM VALUES (1) tab(col) UNION ALL SELECT
theta_sketch_agg(col, 20) as sketch FROM VALUES (1) tab(col));
@@ -507,6 +514,10 @@ case class ThetaUnionAgg(
usage = """
_FUNC_(expr) - Returns the ThetaSketch's Compact binary representation
by intersecting all the Theta sketches in the input column.""",
+ arguments = """
+ Arguments:
+ * expr - A binary expression of Compact ThetaSketch representations to
intersect.
+ """,
examples = """
Examples:
> SELECT theta_sketch_estimate(_FUNC_(sketch)) FROM (SELECT
theta_sketch_agg(col) as sketch FROM VALUES (1) tab(col) UNION ALL SELECT
theta_sketch_agg(col, 20) as sketch FROM VALUES (1) tab(col));
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
index 03bb84c2623e..d0d7507e3250 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
@@ -502,6 +502,13 @@ object Add {
@ExpressionDescription(
usage = "expr1 _FUNC_ expr2 - Returns `expr1`-`expr2`.",
+ arguments = """
+ Arguments:
+ * expr1 - The minuend.
+ An expression that evaluates to a numeric, interval, date,
timestamp, or time.
+ * expr2 - The subtrahend.
+ An expression that evaluates to a numeric, interval, date,
timestamp, or time.
+ """,
examples = """
Examples:
> SELECT 2 _FUNC_ 1;
@@ -1313,6 +1320,12 @@ object Pmod {
*/
@ExpressionDescription(
usage = "_FUNC_(expr, ...) - Returns the least value of all parameters,
skipping null values.",
+ arguments = """
+ Arguments:
+ * expr - An expression of any orderable type. At least two expressions
must be given,
+ and all arguments must share a common type. Null values are skipped;
the result is
+ null only when all arguments are null.
+ """,
examples = """
Examples:
> SELECT _FUNC_(10, 9, 2, 4, 3);
@@ -1402,6 +1415,12 @@ case class Least(children: Seq[Expression]) extends
ComplexTypeMergingExpression
*/
@ExpressionDescription(
usage = "_FUNC_(expr, ...) - Returns the greatest value of all parameters,
skipping null values.",
+ arguments = """
+ Arguments:
+ * expr - An expression of any orderable type. At least two expressions
must be given,
+ and all arguments must share a common type. Null values are skipped;
the result is
+ null only when all arguments are null.
+ """,
examples = """
Examples:
> SELECT _FUNC_(10, 9, 2, 4, 3);
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala
index aea04913e468..cb553aeb96ff 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala
@@ -40,6 +40,13 @@ import org.apache.spark.util.Utils
usage = """
_FUNC_(child, jsonFormatSchema, options) - Converts a binary Avro value
into a Catalyst value.
""",
+ arguments = """
+ Arguments:
+ * child - A binary value containing Avro-encoded data.
+ * jsonFormatSchema - A constant string with the Avro schema in JSON
format used to
+ interpret the data.
+ * options - A constant map of string keys and values holding conversion
options.
+ """,
examples = """
Examples:
> SELECT _FUNC_(s, '{"type": "record", "name": "struct", "fields": [{
"name": "u", "type": ["int","string"] }]}', map()) IS NULL AS result FROM
(SELECT NAMED_STRUCT('u', NAMED_STRUCT('member0', member0, 'member1', member1))
AS s FROM VALUES (1, NULL), (NULL, 'a') tab(member0, member1));
@@ -141,6 +148,12 @@ case class FromAvro(child: Expression, jsonFormatSchema:
Expression, options: Ex
_FUNC_(child[, jsonFormatSchema]) - Converts a Catalyst binary input value
into its
corresponding Avro format result.
""",
+ arguments = """
+ Arguments:
+ * child - A value to encode into Avro binary format.
+ * jsonFormatSchema - An optional constant string with the Avro schema in
JSON format
+ used to encode the value. If omitted, the schema is inferred from
the input type.
+ """,
examples = """
Examples:
> SELECT _FUNC_(s, '{"type": "record", "name": "struct", "fields": [{
"name": "u", "type": ["int","string"] }]}') IS NULL FROM (SELECT NULL AS s);
@@ -215,6 +228,11 @@ case class ToAvro(child: Expression, jsonFormatSchema:
Expression)
usage = """
_FUNC_(jsonFormatSchema, options) - Returns schema in the DDL format of
the avro schema in JSON string format.
""",
+ arguments = """
+ Arguments:
+ * jsonFormatSchema - A constant string with the Avro schema in JSON
format.
+ * options - A constant map of string keys and values holding conversion
options.
+ """,
examples = """
Examples:
> SELECT _FUNC_('{"type": "record", "name": "struct", "fields":
[{"name": "u", "type": ["int", "string"]}]}', map());
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/bitmapExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/bitmapExpressions.scala
index 1f943ed7edb0..87c04d229e91 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/bitmapExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/bitmapExpressions.scala
@@ -106,6 +106,11 @@ case class BitmapBitPosition(child: Expression)
@ExpressionDescription(
usage = "_FUNC_(child) - Returns the number of set bits in the child
bitmap.",
+ arguments = """
+ Arguments:
+ * child - The bitmap whose set bits are counted. An expression that
evaluates to a binary
+ bitmap, typically produced by bitmap_construct_agg().
+ """,
examples = """
Examples:
> SELECT _FUNC_(X '1010');
@@ -250,6 +255,11 @@ case class BitmapConstructAgg(child: Expression,
_FUNC_(child) - Returns a bitmap that is the bitwise OR of all of the
bitmaps from the child
expression. The input should be bitmaps created from
bitmap_construct_agg().
""",
+ arguments = """
+ Arguments:
+ * child - The expression whose bitmap values are combined with a bitwise
OR.
+ An expression that evaluates to a binary bitmap created from
bitmap_construct_agg().
+ """,
// scalastyle:off line.size.limit
examples = """
Examples:
@@ -343,6 +353,11 @@ case class BitmapOrAgg(child: Expression,
_FUNC_(child) - Returns a bitmap that is the bitwise AND of all of the
bitmaps from the child
expression. The input should be bitmaps created from
bitmap_construct_agg().
""",
+ arguments = """
+ Arguments:
+ * child - The expression whose bitmap values are combined with a bitwise
AND.
+ An expression that evaluates to a binary bitmap created from
bitmap_construct_agg().
+ """,
// scalastyle:off line.size.limit
examples = """
Examples:
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
index bae2e0f6b383..4a842f2c27f9 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
@@ -191,6 +191,10 @@ case class ArraySize(child: Expression)
*/
@ExpressionDescription(
usage = "_FUNC_(map) - Returns an unordered array containing the keys of the
map.",
+ arguments = """
+ Arguments:
+ * map - A map expression whose keys are returned as an array.
+ """,
examples = """
Examples:
> SELECT _FUNC_(map(1, 'a', 2, 'b'));
@@ -226,6 +230,11 @@ case class MapKeys(child: Expression)
*/
@ExpressionDescription(
usage = "_FUNC_(map, key) - Returns true if the map contains the key.",
+ arguments = """
+ Arguments:
+ * map - A map expression to search.
+ * key - A key to look for. Its type must match, or be coercible to, the
map's key type.
+ """,
examples = """
Examples:
> SELECT _FUNC_(map(1, 'a', 2, 'b'), 1);
@@ -488,6 +497,10 @@ object ArraysZip {
*/
@ExpressionDescription(
usage = "_FUNC_(map) - Returns an unordered array containing the values of
the map.",
+ arguments = """
+ Arguments:
+ * map - A map expression whose values are returned as an array.
+ """,
examples = """
Examples:
> SELECT _FUNC_(map(1, 'a', 2, 'b'));
@@ -522,6 +535,10 @@ case class MapValues(child: Expression)
*/
@ExpressionDescription(
usage = "_FUNC_(map) - Returns an unordered array of all entries in the
given map.",
+ arguments = """
+ Arguments:
+ * map - A map expression whose entries are returned as an array of
key-value structs.
+ """,
examples = """
Examples:
> SELECT _FUNC_(map(1, 'a', 2, 'b'));
@@ -697,6 +714,11 @@ case class MapEntries(child: Expression)
*/
@ExpressionDescription(
usage = "_FUNC_(map, ...) - Returns the union of all the given maps",
+ arguments = """
+ Arguments:
+ * map - A map expression. There can be one or more of them, and all must
share
+ compatible key and value types.
+ """,
examples = """
Examples:
> SELECT _FUNC_(map(1, 'a', 2, 'b'), map(3, 'c'));
@@ -825,6 +847,10 @@ case class MapConcat(children: Seq[Expression])
*/
@ExpressionDescription(
usage = "_FUNC_(arrayOfEntries) - Returns a map created from the given array
of entries.",
+ arguments = """
+ Arguments:
+ * arrayOfEntries - An array of two-field key-value structs from which
the map is built.
+ """,
examples = """
Examples:
> SELECT _FUNC_(array(struct(1, 'a'), struct(2, 'b')));
@@ -1621,6 +1647,12 @@ case class ArrayContains(left: Expression, right:
Expression)
@ExpressionDescription(
usage = "_FUNC_(array, value) - Return index (0-based) of the search value,
" +
"if it is contained in the array; otherwise, (-<insertion point> - 1).",
+ arguments = """
+ Arguments:
+ * array - A sorted array expression to search in.
+ * value - The value to search for. Its type must match, or be coercible
to, the
+ array's element type.
+ """,
examples = """
Examples:
> SELECT _FUNC_(array(1, 2, 3), 2);
@@ -3022,6 +3054,11 @@ case class TryElementAt(left: Expression, right:
Expression, replacement: Expres
*/
@ExpressionDescription(
usage = "_FUNC_(col1, col2, ..., colN) - Returns the concatenation of col1,
col2, ..., colN.",
+ arguments = """
+ Arguments:
+ * colN - An expression to concatenate. There can be one or more of them,
and all must be
+ of the same type: strings, binaries, or arrays.
+ """,
examples = """
Examples:
> SELECT _FUNC_('Spark', 'SQL');
@@ -3243,6 +3280,11 @@ case class Concat(children: Seq[Expression]) extends
ComplexTypeMergingExpressio
*/
@ExpressionDescription(
usage = "_FUNC_(arrayOfArrays) - Transforms an array of arrays into a single
array.",
+ arguments = """
+ Arguments:
+ * arrayOfArrays - An array whose elements are themselves arrays; they
are concatenated
+ in order into one array.
+ """,
examples = """
Examples:
> SELECT _FUNC_(array(array(1, 2), array(3, 4)));
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
index 122272ba4333..8629f900ac43 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
@@ -55,6 +55,11 @@ trait NoThrow
*/
@ExpressionDescription(
usage = "_FUNC_(expr, ...) - Returns an array with the given elements.",
+ arguments = """
+ Arguments:
+ * expr - An expression of any type to include as an array element. Zero
or more
+ expressions can be given, and they must share a common type.
+ """,
examples = """
Examples:
> SELECT _FUNC_(1, 2, 3);
@@ -179,6 +184,14 @@ private [sql] object GenArrayData {
*/
@ExpressionDescription(
usage = "_FUNC_(key0, value0, key1, value1, ...) - Creates a map with the
given key/value pairs.",
+ arguments = """
+ Arguments:
+ * keyN - An expression for the N-th map key. Keys must not be NULL and
all keys must
+ share a common type.
+ * valueN - An expression for the value paired with keyN. All values must
share a common
+ type. Keys and values are supplied as alternating arguments, so the
total number of
+ arguments must be even.
+ """,
examples = """
Examples:
> SELECT _FUNC_(1.0, '2', 3.0, '4');
@@ -423,7 +436,12 @@ object CreateStruct {
null,
"struct",
"_FUNC_(col1, col2, col3, ...) - Creates a struct with the given field
values.",
- "",
+ """
+ | Arguments:
+ | * colN - The field values of the struct. There can be zero or
more of them,
+ | each an expression of any type. Field names are assigned as
`colN` by
+ | default unless the value is a named expression.
+ | """.stripMargin,
"""
| Examples:
| > SELECT _FUNC_(1, 2, 3);
@@ -446,6 +464,13 @@ object CreateStruct {
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(name1, val1, name2, val2, ...) - Creates a struct with the
given field names and values.",
+ arguments = """
+ Arguments:
+ * nameN - A STRING literal giving the name of the N-th struct field. It
must not be NULL.
+ * valN - An expression of any type providing the value for the field
named nameN. Names
+ and values are supplied as alternating arguments, so the total
number of arguments
+ must be even.
+ """,
examples = """
Examples:
> SELECT _FUNC_("a", 1, "b", 2, "c", 3);
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala
index 621f02ca18b8..24dd3623cd13 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala
@@ -33,6 +33,12 @@ import org.apache.spark.util.ArrayImplicits._
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(expr1, expr2, expr3) - If `expr1` evaluates to true, then
returns `expr2`; otherwise returns `expr3`.",
+ arguments = """
+ Arguments:
+ * expr1 - A boolean expression evaluated as the condition.
+ * expr2 - The expression returned when `expr1` evaluates to true.
+ * expr3 - The expression returned when `expr1` evaluates to false or
null.
+ """,
examples = """
Examples:
> SELECT _FUNC_(1 < 2, 'a', 'b');
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/csvExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/csvExpressions.scala
index 5110aab9c6f0..ae10f0245141 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/csvExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/csvExpressions.scala
@@ -40,6 +40,14 @@ import org.apache.spark.unsafe.types.UTF8String
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(csvStr, schema[, options]) - Returns a struct value with the
given `csvStr` and `schema`.",
+ arguments = """
+ Arguments:
+ * csvStr - A string expression of a single CSV record.
+ * schema - A string literal or invocation of `schema_of_csv` describing
the schema.
+ * options - An optional map literal of string key-value pairs specifying
CSV parsing
+ options controlling how `csvStr` is parsed. Accepts the same options
as the CSV data
+ source.
+ """,
examples = """
Examples:
> SELECT _FUNC_('1, 0.8', 'a INT, b DOUBLE');
@@ -133,6 +141,12 @@ case class CsvToStructs(
*/
@ExpressionDescription(
usage = "_FUNC_(csv[, options]) - Returns schema in the DDL format of CSV
string.",
+ arguments = """
+ Arguments:
+ * csv - A foldable string expression of a single CSV record.
+ * options - An optional map literal of string key-value pairs specifying
CSV parsing
+ options that control schema inference. Accepts the same options as
the CSV data source.
+ """,
examples = """
Examples:
> SELECT _FUNC_('1,abc');
@@ -205,6 +219,13 @@ case class SchemaOfCsv(
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(expr[, options]) - Returns a CSV string with a given struct
value",
+ arguments = """
+ Arguments:
+ * expr - A struct expression to convert into a CSV string.
+ * options - An optional map literal of string key-value pairs specifying
CSV generation
+ options controlling how the struct is rendered. Accepts the same
options as the CSV data
+ source.
+ """,
examples = """
Examples:
> SELECT _FUNC_(named_struct('a', 1, 'b', 2));
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datasketchesExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datasketchesExpressions.scala
index a9baf473c822..743dd27c9240 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datasketchesExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datasketchesExpressions.scala
@@ -29,6 +29,11 @@ import org.apache.spark.sql.types.{AbstractDataType,
BinaryType, BooleanType, Da
usage = """
_FUNC_(expr) - Returns the estimated number of unique values given the
binary representation
of a Datasketches HllSketch. """,
+ arguments = """
+ Arguments:
+ * expr - A binary expression holding the serialized representation of a
Datasketches
+ HllSketch.
+ """,
examples = """
Examples:
> SELECT _FUNC_(hll_sketch_agg(col)) FROM VALUES (1), (1), (2), (2), (3)
tab(col);
@@ -70,6 +75,15 @@ case class HllSketchEstimate(child: Expression)
Datasketches HllSketch objects, using a Datasketches Union object. Set
allowDifferentLgConfigK to true to allow unions of sketches with different
lgConfigK values (defaults to false). """,
+ arguments = """
+ Arguments:
+ * first - A binary expression holding the serialized representation of a
Datasketches
+ HllSketch.
+ * second - A binary expression holding the serialized representation of
a Datasketches
+ HllSketch.
+ * allowDifferentLgConfigK - A boolean. Set to true to allow unions of
sketches with
+ different lgConfigK values. Defaults to false.
+ """,
examples = """
Examples:
> SELECT hll_sketch_estimate(_FUNC_(hll_sketch_agg(col1),
hll_sketch_agg(col2))) FROM VALUES (1, 4), (1, 4), (2, 5), (2, 5), (3, 6)
tab(col1, col2);
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
index 0b6dcb0cd875..992f58d09c3a 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
@@ -1126,6 +1126,10 @@ case class UnixMicros(child: Expression) extends
TimestampToLongBase {
// scalastyle:off line.contains.tab
@ExpressionDescription(
usage = "_FUNC_(timestamp) - Returns the number of nanoseconds since
1970-01-01 00:00:00 UTC.",
+ arguments = """
+ Arguments:
+ * timestamp - A nanosecond-precision timestamp value (TIMESTAMP_LTZ or
TIMESTAMP_NTZ).
+ """,
examples = """
Examples:
> SET spark.sql.timestampNanosTypes.enabled=true;
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala
index de8d40cd4299..3f2e838c1903 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala
@@ -143,6 +143,13 @@ case class UserDefinedGenerator(
// scalastyle:off line.size.limit line.contains.tab
@ExpressionDescription(
usage = "_FUNC_(n, expr1, ..., exprk) - Separates `expr1`, ..., `exprk` into
`n` rows. Uses column names col0, col1, etc. by default unless specified
otherwise.",
+ arguments = """
+ Arguments:
+ * n - The number of rows to separate the expressions into. Must be a
+ positive integer literal.
+ * exprN - The expressions to separate into rows. Their values are laid
out
+ row-major across the `n` output rows.
+ """,
examples = """
Examples:
> SELECT _FUNC_(2, 1, 2, 3);
@@ -437,6 +444,10 @@ trait ExplodeGeneratorBuilderBase extends GeneratorBuilder
{
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(expr) - Separates the elements of array `expr` into multiple
rows, or the elements of map `expr` into multiple rows and columns. Unless
specified otherwise, uses the default column name `col` for elements of the
array or `key` and `value` for the elements of the map.",
+ arguments = """
+ Arguments:
+ * expr - An array or map expression whose elements are separated into
rows.
+ """,
examples = """
Examples:
> SELECT _FUNC_(array(10, 20));
@@ -522,6 +533,11 @@ trait PosExplodeGeneratorBuilderBase extends
GeneratorBuilder {
// scalastyle:off line.size.limit line.contains.tab
@ExpressionDescription(
usage = "_FUNC_(expr) - Separates the elements of array `expr` into multiple
rows with positions, or the elements of map `expr` into multiple rows and
columns with positions. Unless specified otherwise, uses the column name `pos`
for position, `col` for elements of the array or `key` and `value` for elements
of the map.",
+ arguments = """
+ Arguments:
+ * expr - An array or map expression whose elements are separated into
rows,
+ each paired with its position.
+ """,
examples = """
Examples:
> SELECT _FUNC_(array(10,20));
@@ -643,6 +659,11 @@ trait InlineGeneratorBuilderBase extends GeneratorBuilder {
// scalastyle:off line.size.limit line.contains.tab
@ExpressionDescription(
usage = "_FUNC_(expr) - Explodes an array of structs into a table. Uses
column names col1, col2, etc. by default unless specified otherwise.",
+ arguments = """
+ Arguments:
+ * expr - An array-of-structs expression whose struct fields become the
+ columns of each generated row.
+ """,
examples = """
Examples:
> SELECT _FUNC_(array(struct(1, 'a'), struct(2, 'b')));
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/grouping.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/grouping.scala
index 0a12735b38da..adb26487b1bf 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/grouping.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/grouping.scala
@@ -196,6 +196,11 @@ object GroupingSets {
_FUNC_(col) - indicates whether a specified column in a GROUP BY is
aggregated or
not, returns 1 for aggregated or 0 for not aggregated in the result
set.",
""",
+ arguments = """
+ Arguments:
+ * col - A grouping column referenced in the GROUP BY clause. It must
exactly match one
+ of the grouping expressions of the query.
+ """,
examples = """
Examples:
> SELECT name, _FUNC_(name), sum(age) FROM VALUES (2, 'Alice'), (5,
'Bob') people(age, name) GROUP BY cube(name);
@@ -229,6 +234,12 @@ case class Grouping(child: Expression) extends Expression
with Unevaluable
_FUNC_([col1[, col2 ..]]) - returns the level of grouping, equals to
`(grouping(c1) << (n-1)) + (grouping(c2) << (n-2)) + ... + grouping(cn)`
""",
+ arguments = """
+ Arguments:
+ * colN - An optional grouping column referenced in the GROUP BY clause.
Zero or more
+ columns can be given; when provided, they must match the grouping
columns exactly,
+ and when omitted all grouping columns are used.
+ """,
examples = """
Examples:
> SELECT name, _FUNC_(), sum(age), avg(height) FROM VALUES (2, 'Alice',
165), (5, 'Bob', 180) people(age, name, height) GROUP BY cube(name, height);
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala
index 085586041b07..82c273804d23 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala
@@ -762,6 +762,11 @@ abstract class InterpretedHashFunction {
*/
@ExpressionDescription(
usage = "_FUNC_(expr1, expr2, ...) - Returns a hash value of the arguments.",
+ arguments = """
+ Arguments:
+ * exprN - The values to hash. There can be one or more of them, each an
+ expression of any data type.
+ """,
examples = """
Examples:
> SELECT _FUNC_('Spark', array(123), 2);
@@ -833,6 +838,11 @@ case class CollationAwareMurmur3Hash(children:
Seq[Expression], seed: Int)
@ExpressionDescription(
usage = "_FUNC_(expr1, expr2, ...) - Returns a 64-bit hash value of the
arguments. " +
"Hash seed is 42.",
+ arguments = """
+ Arguments:
+ * exprN - The values to hash. There can be one or more of them, each an
+ expression of any data type.
+ """,
examples = """
Examples:
> SELECT _FUNC_('Spark', array(123), 2);
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala
index 524e27321f4f..cc34421b6230 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala
@@ -658,6 +658,12 @@ object ArraySort {
*/
@ExpressionDescription(
usage = "_FUNC_(expr, func) - Filters entries in a map using the function.",
+ arguments = """
+ Arguments:
+ * expr - A map expression.
+ * func - A lambda function `(k, v) -> boolean` that returns whether the
entry with
+ key `k` and value `v` should be kept.
+ """,
examples = """
Examples:
> SELECT _FUNC_(map(1, 0, 2, 2, 3, -1), (k, v) -> k > v);
@@ -1352,6 +1358,12 @@ case class ArrayAggregate(
*/
@ExpressionDescription(
usage = "_FUNC_(expr, func) - Transforms elements in a map using the
function.",
+ arguments = """
+ Arguments:
+ * expr - A map expression.
+ * func - A lambda function `(k, v) -> newKey` producing the transformed
key from the
+ entry with key `k` and value `v`. The values are kept unchanged.
+ """,
examples = """
Examples:
> SELECT _FUNC_(map_from_arrays(array(1, 2, 3), array(1, 2, 3)), (k, v)
-> k + 1);
@@ -1412,6 +1424,12 @@ case class TransformKeys(
*/
@ExpressionDescription(
usage = "_FUNC_(expr, func) - Transforms values in the map using the
function.",
+ arguments = """
+ Arguments:
+ * expr - A map expression.
+ * func - A lambda function `(k, v) -> newValue` producing the
transformed value from
+ the entry with key `k` and value `v`. The keys are kept unchanged.
+ """,
examples = """
Examples:
> SELECT _FUNC_(map_from_arrays(array(1, 2, 3), array(1, 2, 3)), (k, v)
-> v + 1);
@@ -1472,6 +1490,14 @@ case class TransformValues(
NULL will be passed as the value for the missing key. If an input map
contains duplicated
keys, only the first entry of the duplicated key is passed into the
lambda function.
""",
+ arguments = """
+ Arguments:
+ * map1 - The first map expression.
+ * map2 - The second map expression.
+ * function - A lambda function `(k, v1, v2) -> newValue` that produces
the merged value
+ for key `k`, where `v1` and `v2` are the values from `map1` and
`map2` respectively
+ (NULL when the key is missing from that map).
+ """,
examples = """
Examples:
> SELECT _FUNC_(map(1, 'a', 2, 'b'), map(1, 'x', 2, 'y'), (k, v1, v2) ->
concat(v1, v2));
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
index d8131c3afb42..1621ec9ba9ac 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
@@ -244,6 +244,12 @@ case class MultiGetJsonObject(
// scalastyle:off line.size.limit line.contains.tab
@ExpressionDescription(
usage = "_FUNC_(jsonStr, p1, p2, ..., pn) - Returns a tuple like the
function get_json_object, but it takes multiple names. All the input parameters
and output column types are string.",
+ arguments = """
+ Arguments:
+ * jsonStr - A JSON string to extract fields from.
+ * pN - The field names to extract. Each name yields one output column
with
+ the corresponding field value.
+ """,
examples = """
Examples:
> SELECT _FUNC_('{"a":1, "b":2}', 'a', 'b');
@@ -346,6 +352,14 @@ case class JsonTuple(children: Seq[Expression])
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(jsonStr, schema[, options]) - Returns a struct value with
the given `jsonStr` and `schema`.",
+ arguments = """
+ Arguments:
+ * jsonStr - A JSON string to parse.
+ * schema - The schema to use when parsing the JSON string, given as a DDL
+ formatted string or a schema expression.
+ * options - Optional. A map of string key-value pairs that control how
the
+ JSON is parsed. By default no options are set.
+ """,
examples = """
Examples:
> SELECT _FUNC_('{"a":1, "b":0.8}', 'a INT, b DOUBLE');
@@ -549,6 +563,12 @@ case class StructsToJson(
*/
@ExpressionDescription(
usage = "_FUNC_(json[, options]) - Returns schema in the DDL format of JSON
string.",
+ arguments = """
+ Arguments:
+ * json - A JSON string whose schema is inferred.
+ * options - Optional. A map of string key-value pairs that control how
the
+ JSON is parsed. By default no options are set.
+ """,
examples = """
Examples:
> SELECT _FUNC_('[{"col":0}]');
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala
index e80277666109..fa456d8955bf 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala
@@ -364,6 +364,10 @@ case class SparkVersion()
@ExpressionDescription(
usage = """_FUNC_(expr) - Return DDL-formatted type string for the data type
of the input.""",
+ arguments = """
+ Arguments:
+ * expr - An expression of any type whose data type is returned as a
DDL-formatted string.
+ """,
examples = """
Examples:
> SELECT _FUNC_(1);
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala
index eb8e67eca3f1..db77d6b82b9f 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala
@@ -41,6 +41,11 @@ import org.apache.spark.sql.types._
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(expr1, expr2, ...) - Returns the first non-null argument if
exists. Otherwise, null.",
+ arguments = """
+ Arguments:
+ * exprN - An expression of any type. All arguments must share a common
type.
+ Arguments are evaluated in order and the first non-null value is
returned.
+ """,
examples = """
Examples:
> SELECT _FUNC_(NULL, 1, NULL);
@@ -252,6 +257,10 @@ case class NullIfZero(input: Expression, replacement:
Expression)
@ExpressionDescription(
usage = "_FUNC_(expr) - Returns zero if `expr` is equal to null, or `expr`
otherwise.",
+ arguments = """
+ Arguments:
+ * expr - An expression. Zero is returned when it is null, otherwise
`expr` is returned.
+ """,
examples = """
Examples:
> SELECT _FUNC_(NULL);
@@ -273,6 +282,11 @@ case class ZeroIfNull(input: Expression, replacement:
Expression)
@ExpressionDescription(
usage = "_FUNC_(expr1, expr2) - Returns `expr2` if `expr1` is null, or
`expr1` otherwise.",
+ arguments = """
+ Arguments:
+ * expr1 - An expression. Returned when it is not null.
+ * expr2 - The value returned when `expr1` is null.
+ """,
examples = """
Examples:
> SELECT _FUNC_(NULL, array('2'));
@@ -297,6 +311,12 @@ case class Nvl(left: Expression, right: Expression,
replacement: Expression)
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(expr1, expr2, expr3) - Returns `expr2` if `expr1` is not
null, or `expr3` otherwise.",
+ arguments = """
+ Arguments:
+ * expr1 - An expression tested for nullability.
+ * expr2 - The value returned when `expr1` is not null.
+ * expr3 - The value returned when `expr1` is null.
+ """,
examples = """
Examples:
> SELECT _FUNC_(NULL, 2, 1);
@@ -464,6 +484,10 @@ case class NaNvl(left: Expression, right: Expression)
*/
@ExpressionDescription(
usage = "_FUNC_(expr) - Returns true if `expr` is null, or false otherwise.",
+ arguments = """
+ Arguments:
+ * expr - An expression of any type to test for nullability.
+ """,
examples = """
Examples:
> SELECT _FUNC_(1);
@@ -498,6 +522,10 @@ case class IsNull(child: Expression) extends
UnaryExpression with Predicate {
*/
@ExpressionDescription(
usage = "_FUNC_(expr) - Returns true if `expr` is not null, or false
otherwise.",
+ arguments = """
+ Arguments:
+ * expr - An expression of any type to test for nullability.
+ """,
examples = """
Examples:
> SELECT _FUNC_(1);
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
index 4ea1fa03173b..b10f941e2d66 100755
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
@@ -280,6 +280,12 @@ case class ConcatWs(children: Seq[Expression])
and `spark.sql.ansi.enabled` is set to false. If `spark.sql.ansi.enabled`
is set to true,
it throws ArrayIndexOutOfBoundsException for invalid indices.
""",
+ arguments = """
+ Arguments:
+ * n - An integer expression giving the 1-based index of the input to
return.
+ * input1, input2, ... - The input expressions to select from. They can
be strings
+ or binary values.
+ """,
examples = """
Examples:
> SELECT _FUNC_(1, 'scala', 'java');
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/thetasketchesExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/thetasketchesExpressions.scala
index 8ac40a3fe2a5..1aa668a9b894 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/thetasketchesExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/thetasketchesExpressions.scala
@@ -28,6 +28,11 @@ import org.apache.spark.sql.types.{AbstractDataType,
BinaryType, DataType, Integ
usage = """
_FUNC_(expr) - Returns the estimated number of unique values
given the binary representation of a Datasketches ThetaSketch. """,
+ arguments = """
+ Arguments:
+ * expr - The binary representation of a Datasketches ThetaSketch.
+ An expression that evaluates to binary.
+ """,
examples = """
Examples:
> SELECT _FUNC_(theta_sketch_agg(col)) FROM VALUES (1), (1), (2), (2),
(3) tab(col);
@@ -66,6 +71,16 @@ case class ThetaSketchEstimate(child: Expression)
Datasketches ThetaSketch objects using a ThetaSketch Union object. Users
can set
lgNomEntries to a value between 4 and 26 to find the union of sketches
with different
union buffer size values (defaults to 12). """,
+ arguments = """
+ Arguments:
+ * first - The binary representation of the first Datasketches
ThetaSketch.
+ An expression that evaluates to binary.
+ * second - The binary representation of the second Datasketches
ThetaSketch.
+ An expression that evaluates to binary.
+ * lgNomEntries - Optional. The log-base-2 of the nominal entries used to
size the
+ union buffer, between 4 and 26. An expression that evaluates to an
integer.
+ Defaults to 12.
+ """,
examples = """
Examples:
> SELECT theta_sketch_estimate(_FUNC_(theta_sketch_agg(col1),
theta_sketch_agg(col2))) FROM VALUES (1, 4), (1, 4), (2, 5), (2, 5), (3, 6)
tab(col1, col2);
@@ -125,6 +140,13 @@ case class ThetaUnion(first: Expression, second:
Expression, third: Expression)
_FUNC_(first, second) - Subtracts two binary representations of
Datasketches ThetaSketch objects from two input columns using a
ThetaSketch AnotB object. """,
+ arguments = """
+ Arguments:
+ * first - The binary representation of the first Datasketches
ThetaSketch.
+ An expression that evaluates to binary.
+ * second - The binary representation of the second Datasketches
ThetaSketch to
+ subtract from the first. An expression that evaluates to binary.
+ """,
examples = """
Examples:
> SELECT theta_sketch_estimate(_FUNC_(theta_sketch_agg(col1),
theta_sketch_agg(col2))) FROM VALUES (5, 4), (1, 4), (2, 5), (2, 5), (3, 1)
tab(col1, col2);
@@ -173,6 +195,13 @@ case class ThetaDifference(first: Expression, second:
Expression)
_FUNC_(first, second) - Intersects two binary representations of
Datasketches ThetaSketch objects from two input columns using a
ThetaSketch Intersect object. """,
+ arguments = """
+ Arguments:
+ * first - The binary representation of the first Datasketches
ThetaSketch.
+ An expression that evaluates to binary.
+ * second - The binary representation of the second Datasketches
ThetaSketch.
+ An expression that evaluates to binary.
+ """,
examples = """
Examples:
> SELECT theta_sketch_estimate(_FUNC_(theta_sketch_agg(col1),
theta_sketch_agg(col2))) FROM VALUES (5, 4), (1, 4), (2, 5), (2, 5), (3, 1)
tab(col1, col2);
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/toFromProtobufSqlFunctions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/toFromProtobufSqlFunctions.scala
index 7d0fa35b3f0f..644695dfabbf 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/toFromProtobufSqlFunctions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/toFromProtobufSqlFunctions.scala
@@ -46,6 +46,18 @@ import org.apache.spark.util.Utils
usage = """
_FUNC_(data, messageName, descFilePath, options) - Converts a binary
Protobuf value into a Catalyst value.
""",
+ arguments = """
+ Arguments:
+ * data - The binary Protobuf value to convert.
+ * messageName - A constant string naming the Protobuf message to look
for in
+ the descriptor file.
+ * descFilePath - Optional. A constant string or binary value with the
+ Protobuf descriptor file, created by `protoc` with
`--descriptor_set_out`
+ and `--include_imports`. If omitted, the message must be resolvable
+ otherwise.
+ * options - Optional. A constant map of string key-value pairs
controlling
+ the conversion. By default no options are set.
+ """,
examples = """
Examples:
> SELECT _FUNC_(s, 'Person', '/path/to/descriptor.desc', map()) IS NULL
AS result FROM (SELECT NAMED_STRUCT('name', name, 'id', id) AS s FROM VALUES
('John Doe', 1), (NULL, 2) tab(name, id));
@@ -190,6 +202,17 @@ case class FromProtobuf(
_FUNC_(child, messageName, descFilePath, options) - Converts a Catalyst
binary input value into its corresponding
Protobuf format result.
""",
+ arguments = """
+ Arguments:
+ * child - The Catalyst input value to convert to Protobuf binary.
+ * messageName - A constant string naming the Protobuf message to
serialize to.
+ * descFilePath - Optional. A constant string or binary value with the
+ Protobuf descriptor file, created by `protoc` with
`--descriptor_set_out`
+ and `--include_imports`. If omitted, the message must be resolvable
+ otherwise.
+ * options - Optional. A constant map of string key-value pairs
controlling
+ the conversion. By default no options are set.
+ """,
examples = """
Examples:
> SELECT _FUNC_(s, 'Person', '/path/to/descriptor.desc',
map('emitDefaultValues', 'true')) IS NULL FROM (SELECT NULL AS s);
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleDifference.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleDifference.scala
index e26354b54f6c..539af821e8e8 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleDifference.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleDifference.scala
@@ -31,6 +31,12 @@ import org.apache.spark.sql.types.{AbstractDataType,
BinaryType, DataType}
_FUNC_(tupleSketch1, tupleSketch2) - Subtracts two binary representations
of Datasketches
TupleSketch objects with double summary data type using a TupleSketch
AnotB object.
Returns elements in the first sketch that are not in the second sketch.
""",
+ arguments = """
+ Arguments:
+ * tupleSketch1 - A binary representation of a TupleSketch with double
summary.
+ * tupleSketch2 - A binary representation of a TupleSketch with double
summary
+ to subtract from the first sketch.
+ """,
examples = """
Examples:
> SELECT
tuple_sketch_estimate_double(_FUNC_(tuple_sketch_agg_double(col1, val1),
tuple_sketch_agg_double(col2, val2))) FROM VALUES (5, 5.0D, 4, 4.0D), (1, 1.0D,
4, 4.0D), (2, 2.0D, 5, 5.0D), (3, 3.0D, 1, 1.0D) tab(col1, val1, col2, val2);
@@ -67,6 +73,12 @@ case class TupleDifferenceDouble(left: Expression, right:
Expression)
_FUNC_(tupleSketch1, tupleSketch2) - Subtracts two binary representations
of Datasketches
TupleSketch objects with integer summary data type using a TupleSketch
AnotB object.
Returns elements in the first sketch that are not in the second sketch.
""",
+ arguments = """
+ Arguments:
+ * tupleSketch1 - A binary representation of a TupleSketch with integer
summary.
+ * tupleSketch2 - A binary representation of a TupleSketch with integer
summary
+ to subtract from the first sketch.
+ """,
examples = """
Examples:
> SELECT
tuple_sketch_estimate_integer(_FUNC_(tuple_sketch_agg_integer(col1, val1),
tuple_sketch_agg_integer(col2, val2))) FROM VALUES (5, 5, 4, 4), (1, 1, 4, 4),
(2, 2, 5, 5), (3, 3, 1, 1) tab(col1, val1, col2, val2);
@@ -103,6 +115,11 @@ case class TupleDifferenceInteger(left: Expression, right:
Expression)
_FUNC_(tupleSketch, thetaSketch) - Subtracts the binary representation of a
Datasketches ThetaSketch from a TupleSketch with double summary data type
using a TupleSketch
AnotB object. Returns elements in the TupleSketch that are not in the
ThetaSketch. """,
+ arguments = """
+ Arguments:
+ * tupleSketch - A binary representation of a TupleSketch with double
summary.
+ * thetaSketch - A binary representation of a ThetaSketch to subtract
from the TupleSketch.
+ """,
examples = """
Examples:
> SELECT
tuple_sketch_estimate_double(_FUNC_(tuple_sketch_agg_double(col1, val1),
theta_sketch_agg(col2))) FROM VALUES (5, 5.0D, 4), (1, 1.0D, 4), (2, 2.0D, 5),
(3, 3.0D, 1) tab(col1, val1, col2);
@@ -139,6 +156,11 @@ case class TupleDifferenceThetaDouble(left: Expression,
right: Expression)
_FUNC_(tupleSketch, thetaSketch) - Subtracts the binary representation of a
Datasketches ThetaSketch from a TupleSketch with integer summary data type
using a TupleSketch
AnotB object. Returns elements in the TupleSketch that are not in the
ThetaSketch. """,
+ arguments = """
+ Arguments:
+ * tupleSketch - A binary representation of a TupleSketch with integer
summary.
+ * thetaSketch - A binary representation of a ThetaSketch to subtract
from the TupleSketch.
+ """,
examples = """
Examples:
> SELECT
tuple_sketch_estimate_integer(_FUNC_(tuple_sketch_agg_integer(col1, val1),
theta_sketch_agg(col2))) FROM VALUES (5, 5, 4), (1, 1, 4), (2, 2, 5), (3, 3, 1)
tab(col1, val1, col2);
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleIntersection.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleIntersection.scala
index 2085b759af09..b6a5cb1d38ce 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleIntersection.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleIntersection.scala
@@ -34,6 +34,15 @@ import org.apache.spark.unsafe.types.UTF8String
_FUNC_(tupleSketch1, tupleSketch2, mode) - Intersects two binary
representations of Datasketches
TupleSketch objects with double summary data type using a TupleSketch
Intersection object.
Users can set mode to 'sum', 'min', 'max', or 'alwaysone' (defaults to
'sum'). """,
+ arguments = """
+ Arguments:
+ * tupleSketch1 - The binary representation of a Datasketches TupleSketch
with a double
+ summary data type.
+ * tupleSketch2 - The binary representation of a Datasketches TupleSketch
with a double
+ summary data type.
+ * mode - The summary combination mode: 'sum', 'min', 'max', or
'alwaysone'
+ (optional, defaults to 'sum').
+ """,
examples = """
Examples:
> SELECT
tuple_sketch_estimate_double(_FUNC_(tuple_sketch_agg_double(col1, val1),
tuple_sketch_agg_double(col2, val2))) FROM VALUES (1, 1.0D, 1, 4.0D), (2, 2.0D,
2, 5.0D), (3, 3.0D, 4, 6.0D) tab(col1, val1, col2, val2);
@@ -80,6 +89,15 @@ case class TupleIntersectionDouble(first: Expression,
second: Expression, third:
_FUNC_(tupleSketch1, tupleSketch2, mode) - Intersects two binary
representations of Datasketches
TupleSketch objects with integer summary data type using a TupleSketch
Intersection object.
Users can set mode to 'sum', 'min', 'max', or 'alwaysone' (defaults to
'sum'). """,
+ arguments = """
+ Arguments:
+ * tupleSketch1 - The binary representation of a Datasketches TupleSketch
with an integer
+ summary data type.
+ * tupleSketch2 - The binary representation of a Datasketches TupleSketch
with an integer
+ summary data type.
+ * mode - The summary combination mode: 'sum', 'min', 'max', or
'alwaysone'
+ (optional, defaults to 'sum').
+ """,
examples = """
Examples:
> SELECT
tuple_sketch_estimate_integer(_FUNC_(tuple_sketch_agg_integer(col1, val1),
tuple_sketch_agg_integer(col2, val2))) FROM VALUES (1, 1, 1, 4), (2, 2, 2, 5),
(3, 3, 4, 6) tab(col1, val1, col2, val2);
@@ -131,6 +149,14 @@ case class TupleIntersectionInteger(first: Expression,
second: Expression, third
assigned a default double summary value based on the mode: 0.0 for 'sum'
mode, +Infinity for
'min' mode, -Infinity for 'max' mode, or 1.0 for 'alwaysone' mode. Users
can set mode to 'sum',
'min', 'max', or 'alwaysone' (defaults to 'sum'). """,
+ arguments = """
+ Arguments:
+ * tupleSketch - The binary representation of a Datasketches TupleSketch
with a double
+ summary data type.
+ * thetaSketch - The binary representation of a Datasketches ThetaSketch.
+ * mode - The summary combination mode: 'sum', 'min', 'max', or
'alwaysone'
+ (optional, defaults to 'sum').
+ """,
examples = """
Examples:
> SELECT
tuple_sketch_estimate_double(_FUNC_(tuple_sketch_agg_double(col1, val1),
theta_sketch_agg(col2))) FROM VALUES (1, 1.0D, 1), (2, 2.0D, 2), (3, 3.0D, 4)
tab(col1, val1, col2);
@@ -182,6 +208,14 @@ case class TupleIntersectionThetaDouble(first: Expression,
second: Expression, t
assigned a default integer summary value based on the mode: 0 for 'sum'
mode, Integer.MAX_VALUE
for 'min' mode, Integer.MIN_VALUE for 'max' mode, or 1 for 'alwaysone'
mode. Users can set mode
to 'sum', 'min', 'max', or 'alwaysone' (defaults to 'sum'). """,
+ arguments = """
+ Arguments:
+ * tupleSketch - The binary representation of a Datasketches TupleSketch
with an integer
+ summary data type.
+ * thetaSketch - The binary representation of a Datasketches ThetaSketch.
+ * mode - The summary combination mode: 'sum', 'min', 'max', or
'alwaysone'
+ (optional, defaults to 'sum').
+ """,
examples = """
Examples:
> SELECT
tuple_sketch_estimate_integer(_FUNC_(tuple_sketch_agg_integer(col1, val1),
theta_sketch_agg(col2))) FROM VALUES (1, 1, 1), (2, 2, 2), (3, 3, 4) tab(col1,
val1, col2);
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleSketchEstimate.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleSketchEstimate.scala
index fcd0a048479b..5ac9504bfa92 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleSketchEstimate.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleSketchEstimate.scala
@@ -27,6 +27,11 @@ import org.apache.spark.sql.types.{AbstractDataType,
BinaryType, DataType, Doubl
_FUNC_(child) - Returns the estimated number of unique values
given the binary representation of a Datasketches TupleSketch. The sketch's
summary type must be a double. """,
+ arguments = """
+ Arguments:
+ * child - A binary value holding a serialized Datasketches TupleSketch
+ with a double summary.
+ """,
examples = """
Examples:
> SELECT _FUNC_(tuple_sketch_agg_double(key, summary)) FROM VALUES (1,
1.0D), (1, 2.0D), (2, 3.0D) tab(key, summary);
@@ -64,6 +69,11 @@ case class TupleSketchEstimateDouble(child: Expression)
_FUNC_(child) - Returns the estimated number of unique values
given the binary representation of a Datasketches TupleSketch. The sketch's
summary type must be an integer. """,
+ arguments = """
+ Arguments:
+ * child - A binary value holding a serialized Datasketches TupleSketch
+ with an integer summary.
+ """,
examples = """
Examples:
> SELECT _FUNC_(tuple_sketch_agg_integer(key, summary)) FROM VALUES (1,
1), (1, 2), (2, 3) tab(key, summary);
@@ -101,6 +111,11 @@ case class TupleSketchEstimateInteger(child: Expression)
_FUNC_(child) - Returns the theta value (sampling rate) from a
Datasketches TupleSketch.
The theta value represents the effective sampling rate of the sketch,
between 0.0 and 1.0.
The sketch's summary type must be a double. """,
+ arguments = """
+ Arguments:
+ * child - A binary value holding a serialized Datasketches TupleSketch
+ with a double summary.
+ """,
examples = """
Examples:
> SELECT _FUNC_(tuple_sketch_agg_double(key, summary)) FROM VALUES (1,
1.0D), (2, 2.0D), (3, 3.0D) tab(key, summary);
@@ -138,6 +153,11 @@ case class TupleSketchThetaDouble(child: Expression)
_FUNC_(child) - Returns the theta value (sampling rate) from a
Datasketches TupleSketch.
The theta value represents the effective sampling rate of the sketch,
between 0.0 and 1.0.
The sketch's summary type must be an integer. """,
+ arguments = """
+ Arguments:
+ * child - A binary value holding a serialized Datasketches TupleSketch
+ with an integer summary.
+ """,
examples = """
Examples:
> SELECT _FUNC_(tuple_sketch_agg_integer(key, summary)) FROM VALUES (1,
1), (2, 2), (3, 3) tab(key, summary);
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleSketchSummary.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleSketchSummary.scala
index a37a03e41e53..625ab10c3119 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleSketchSummary.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleSketchSummary.scala
@@ -34,6 +34,13 @@ import org.apache.spark.unsafe.types.UTF8String
_FUNC_(child, mode) - Aggregates the summary values from a double summary
type
Datasketches TupleSketch. The mode can be 'sum', 'min', 'max', or
'alwaysone'
(defaults to 'sum'). """,
+ arguments = """
+ Arguments:
+ * child - A binary expression holding a serialized double summary type
Datasketches
+ TupleSketch to aggregate.
+ * mode - An optional string aggregation mode: 'sum', 'min', 'max', or
'alwaysone'.
+ Defaults to 'sum'.
+ """,
examples = """
Examples:
> SELECT _FUNC_(tuple_sketch_agg_double(key, summary)) FROM VALUES (1,
1.0D), (1, 2.0D), (2, 3.0D) tab(key, summary);
@@ -98,6 +105,13 @@ case class TupleSketchSummaryDouble(left: Expression,
right: Expression)
_FUNC_(child, mode) - Aggregates the summary values from a integer summary
type
Datasketches TupleSketch. The mode can be 'sum', 'min', 'max', or
'alwaysone'
(defaults to 'sum'). """,
+ arguments = """
+ Arguments:
+ * child - A binary expression holding a serialized integer summary type
Datasketches
+ TupleSketch to aggregate.
+ * mode - An optional string aggregation mode: 'sum', 'min', 'max', or
'alwaysone'.
+ Defaults to 'sum'.
+ """,
examples = """
Examples:
> SELECT _FUNC_(tuple_sketch_agg_integer(key, summary)) FROM VALUES (1,
1), (1, 2), (2, 3) tab(key, summary);
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleUnion.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleUnion.scala
index be04fb20dd30..d174e392bb93 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleUnion.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleUnion.scala
@@ -377,6 +377,15 @@ abstract class TupleUnionBase[S <: Summary]
TupleSketch objects with double summary data type using a TupleSketch
Union object. Users can
set lgNomEntries to a value between 4 and 26 (defaults to 12) and mode to
'sum', 'min', 'max',
or 'alwaysone' (defaults to 'sum'). """,
+ arguments = """
+ Arguments:
+ * tupleSketch1 - The first TupleSketch (double summary) as a binary
value.
+ * tupleSketch2 - The second TupleSketch (double summary) as a binary
value.
+ * lgNomEntries - Optional integer between 4 and 26 setting the log base
2 of the
+ number of nominal entries. Defaults to 12.
+ * mode - Optional summary combining mode, one of 'sum', 'min', 'max', or
'alwaysone'.
+ Defaults to 'sum'.
+ """,
examples = """
Examples:
> SELECT
tuple_sketch_estimate_double(_FUNC_(tuple_sketch_agg_double(col1, val1),
tuple_sketch_agg_double(col2, val2))) FROM VALUES (1, 1.0D, 4, 4.0D), (2, 2.0D,
5, 5.0D), (3, 3.0D, 6, 6.0D) tab(col1, val1, col2, val2);
@@ -407,6 +416,15 @@ object TupleUnionDoubleExpressionBuilder extends
ExpressionBuilder {
TupleSketch objects with integer summary data type using a TupleSketch
Union object. Users can
set lgNomEntries to a value between 4 and 26 (defaults to 12) and mode to
'sum', 'min', 'max',
or 'alwaysone' (defaults to 'sum'). """,
+ arguments = """
+ Arguments:
+ * tupleSketch1 - The first TupleSketch (integer summary) as a binary
value.
+ * tupleSketch2 - The second TupleSketch (integer summary) as a binary
value.
+ * lgNomEntries - Optional integer between 4 and 26 setting the log base
2 of the
+ number of nominal entries. Defaults to 12.
+ * mode - Optional summary combining mode, one of 'sum', 'min', 'max', or
'alwaysone'.
+ Defaults to 'sum'.
+ """,
examples = """
Examples:
> SELECT
tuple_sketch_estimate_integer(_FUNC_(tuple_sketch_agg_integer(col1, val1),
tuple_sketch_agg_integer(col2, val2))) FROM VALUES (1, 1, 4, 4), (2, 2, 5, 5),
(3, 3, 6, 6) tab(col1, val1, col2, val2);
@@ -440,6 +458,15 @@ object TupleUnionIntegerExpressionBuilder extends
ExpressionBuilder {
-Infinity for 'max' mode, or 1.0 for 'alwaysone' mode. Users can set
lgNomEntries to a value
between 4 and 26 (defaults to 12) and mode to 'sum', 'min', 'max', or
'alwaysone' (defaults to
'sum'). """,
+ arguments = """
+ Arguments:
+ * tupleSketch - The TupleSketch (double summary) as a binary value.
+ * thetaSketch - The ThetaSketch as a binary value.
+ * lgNomEntries - Optional integer between 4 and 26 setting the log base
2 of the
+ number of nominal entries. Defaults to 12.
+ * mode - Optional summary combining mode, one of 'sum', 'min', 'max', or
'alwaysone'.
+ Defaults to 'sum'.
+ """,
examples = """
Examples:
> SELECT
tuple_sketch_estimate_double(_FUNC_(tuple_sketch_agg_double(col1, val1),
theta_sketch_agg(col2))) FROM VALUES (1, 1.0D, 4), (2, 2.0D, 5), (3, 3.0D, 6)
tab(col1, val1, col2);
@@ -473,6 +500,15 @@ object TupleUnionThetaDoubleExpressionBuilder extends
ExpressionBuilder {
mode, Integer.MIN_VALUE for 'max' mode, or 1 for 'alwaysone' mode. Users
can set lgNomEntries to
a value between 4 and 26 (defaults to 12) and mode to 'sum', 'min', 'max',
or 'alwaysone'
(defaults to 'sum'). """,
+ arguments = """
+ Arguments:
+ * tupleSketch - The TupleSketch (integer summary) as a binary value.
+ * thetaSketch - The ThetaSketch as a binary value.
+ * lgNomEntries - Optional integer between 4 and 26 setting the log base
2 of the
+ number of nominal entries. Defaults to 12.
+ * mode - Optional summary combining mode, one of 'sum', 'min', 'max', or
'alwaysone'.
+ Defaults to 'sum'.
+ """,
examples = """
Examples:
> SELECT
tuple_sketch_estimate_integer(_FUNC_(tuple_sketch_agg_integer(col1, val1),
theta_sketch_agg(col2))) FROM VALUES (1, 1, 4), (2, 2, 5), (3, 3, 6) tab(col1,
val1, col2);
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala
index 3448bfee62e1..61a67bf92648 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala
@@ -81,6 +81,10 @@ case class ParseJson(child: Expression, failOnError: Boolean
= true)
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(expr) - Check if a variant value is a variant null. Returns
true if and only if the input is a variant null and false otherwise (including
in the case of SQL NULL).",
+ arguments = """
+ Arguments:
+ * expr - A variant value to check.
+ """,
examples = """
Examples:
> SELECT _FUNC_(parse_json('null'));
@@ -122,6 +126,11 @@ case class IsVariantNull(child: Expression) extends
UnaryExpression
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(expr) - Convert a nested input (array/map/struct) into a
variant where maps and structs are converted to variant objects which are
unordered unlike SQL structs. Input maps can only have string keys.",
+ arguments = """
+ Arguments:
+ * expr - A nested value of array, map, or struct type. Maps must have
string keys. Nested
+ values of any type are allowed.
+ """,
examples = """
Examples:
> SELECT _FUNC_(named_struct('a', 1, 'b', 2));
@@ -638,6 +647,13 @@ abstract class
VariantGetExpressionBuilderBase(failOnError: Boolean) extends Exp
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(v, path[, type]) - Extracts a sub-variant from `v` according
to `path`, and then cast the sub-variant to `type`. When `type` is omitted, it
is default to `variant`. Returns null if the path does not exist. Throws an
exception if the cast fails.",
+ arguments = """
+ Arguments:
+ * v - A variant value to extract from.
+ * path - A string literal in JSONPath format that identifies the
sub-variant to extract.
+ * type - An optional string literal naming the SQL type to cast the
extracted sub-variant to.
+ When omitted, it defaults to `variant`.
+ """,
examples = """
Examples:
> SELECT _FUNC_(parse_json('{"a": 1}'), '$.a', 'int');
@@ -660,6 +676,13 @@ object VariantGetExpressionBuilder extends
VariantGetExpressionBuilderBase(true)
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(v, path[, type]) - Extracts a sub-variant from `v` according
to `path`, and then cast the sub-variant to `type`. When `type` is omitted, it
is default to `variant`. Returns null if the path does not exist or the cast
fails.",
+ arguments = """
+ Arguments:
+ * v - A variant value to extract from.
+ * path - A string literal in JSONPath format that identifies the
sub-variant to extract.
+ * type - An optional string literal naming the SQL type to cast the
extracted sub-variant to.
+ When omitted, it defaults to `variant`.
+ """,
examples = """
Examples:
> SELECT _FUNC_(parse_json('{"a": 1}'), '$.a', 'int');
@@ -1584,6 +1607,10 @@ object VariantExplode {
@ExpressionDescription(
usage = "_FUNC_(v) - Returns schema in the SQL format of a variant.",
+ arguments = """
+ Arguments:
+ * v - A variant value whose schema is returned.
+ """,
examples = """
Examples:
> SELECT _FUNC_(parse_json('null'));
@@ -1699,6 +1726,10 @@ object SchemaOfVariant {
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(v) - Returns the merged schema in the SQL format of a
variant column.",
+ arguments = """
+ Arguments:
+ * v - A variant column whose per-row schemas are merged into a single
schema.
+ """,
examples = """
Examples:
> SELECT _FUNC_(parse_json(j)) FROM VALUES ('1'), ('2'), ('3') AS tab(j);
@@ -1763,6 +1794,10 @@ case class SchemaOfVariantAgg(
@ExpressionDescription(
usage = "_FUNC_(v) - Returns true if the variant is valid, false if it is
malformed, " +
"NULL if `v` is NULL.",
+ arguments = """
+ Arguments:
+ * v - A variant value to validate.
+ """,
examples = """
Examples:
> SELECT _FUNC_(parse_json('null'));
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/vectorExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/vectorExpressions.scala
index e65fae3a2bc2..d730c8f0be7b 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/vectorExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/vectorExpressions.scala
@@ -43,6 +43,12 @@ import org.apache.spark.unsafe.Platform
_FUNC_(array1, array2) - Returns the cosine similarity between two float
vectors.
The vectors must have the same dimension.
""",
+ arguments = """
+ Arguments:
+ * array1 - An ARRAY of FLOAT values representing the first vector.
+ * array2 - An ARRAY of FLOAT values representing the second vector. It
must have the
+ same dimension as array1.
+ """,
examples = """
Examples:
> SELECT _FUNC_(array(1.0F, 2.0F, 3.0F), array(4.0F, 5.0F, 6.0F));
@@ -101,6 +107,12 @@ case class VectorCosineSimilarity(left: Expression, right:
Expression)
_FUNC_(array1, array2) - Returns the inner product (dot product) between
two float vectors.
The vectors must have the same dimension.
""",
+ arguments = """
+ Arguments:
+ * array1 - An ARRAY of FLOAT values representing the first vector.
+ * array2 - An ARRAY of FLOAT values representing the second vector. It
must have the
+ same dimension as array1.
+ """,
examples = """
Examples:
> SELECT _FUNC_(array(1.0F, 2.0F, 3.0F), array(4.0F, 5.0F, 6.0F));
@@ -159,6 +171,12 @@ case class VectorInnerProduct(left: Expression, right:
Expression)
_FUNC_(array1, array2) - Returns the Euclidean (L2) distance between two
float vectors.
The vectors must have the same dimension.
""",
+ arguments = """
+ Arguments:
+ * array1 - An ARRAY of FLOAT values representing the first vector.
+ * array2 - An ARRAY of FLOAT values representing the second vector. It
must have the
+ same dimension as array1.
+ """,
examples = """
Examples:
> SELECT _FUNC_(array(1.0F, 2.0F, 3.0F), array(4.0F, 5.0F, 6.0F));
@@ -218,6 +236,12 @@ case class VectorL2Distance(left: Expression, right:
Expression)
Degree defaults to 2.0 (Euclidean norm) if unspecified. Supported values:
1.0 (L1 norm),
2.0 (L2 norm), float('inf') (infinity norm).
""",
+ arguments = """
+ Arguments:
+ * vector - An ARRAY of FLOAT values representing the vector.
+ * degree - A FLOAT specifying the norm degree. Defaults to 2.0 when
omitted. Supported
+ values are 1.0 (L1 norm), 2.0 (L2 norm), and float('inf') (infinity
norm).
+ """,
examples = """
Examples:
> SELECT _FUNC_(array(3.0F, 4.0F), 2.0F);
@@ -284,6 +308,13 @@ case class VectorNorm(vector: Expression, degree:
Expression)
Degree defaults to 2.0 (Euclidean norm) if unspecified. Supported values:
1.0 (L1 norm),
2.0 (L2 norm), float('inf') (infinity norm).
""",
+ arguments = """
+ Arguments:
+ * vector - An ARRAY of FLOAT values representing the vector to normalize.
+ * degree - A FLOAT specifying the norm degree used for normalization.
Defaults to 2.0
+ when omitted. Supported values are 1.0 (L1 norm), 2.0 (L2 norm), and
float('inf')
+ (infinity norm).
+ """,
examples = """
Examples:
> SELECT _FUNC_(array(3.0F, 4.0F), 2.0F);
@@ -555,6 +586,11 @@ trait VectorAggregateBase extends ImperativeAggregate
_FUNC_(array) - Returns the element-wise mean of float vectors in a group.
All vectors must have the same dimension.
""",
+ arguments = """
+ Arguments:
+ * array - An ARRAY of FLOAT values representing a vector. All vectors
aggregated in the
+ group must have the same dimension.
+ """,
examples = """
Examples:
> SELECT _FUNC_(col) FROM VALUES (array(1.0F, 2.0F)), (array(3.0F,
4.0F)) AS tab(col);
@@ -635,6 +671,11 @@ case class VectorAvg(
_FUNC_(array) - Returns the element-wise sum of float vectors in a group.
All vectors must have the same dimension.
""",
+ arguments = """
+ Arguments:
+ * array - An ARRAY of FLOAT values representing a vector. All vectors
aggregated in the
+ group must have the same dimension.
+ """,
examples = """
Examples:
> SELECT _FUNC_(col) FROM VALUES (array(1.0F, 2.0F)), (array(3.0F,
4.0F)) AS tab(col);
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xmlExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xmlExpressions.scala
index 5a91d3720375..f75536c5475a 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xmlExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xmlExpressions.scala
@@ -38,6 +38,13 @@ import org.apache.spark.unsafe.types.UTF8String
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(xmlStr, schema[, options]) - Returns a struct value with the
given `xmlStr` and `schema`.",
+ arguments = """
+ Arguments:
+ * xmlStr - A string expression containing a single XML record to parse.
+ * schema - The schema of the output struct, as a DDL-formatted string or
a schema
+ expression.
+ * options - Optional. A map of key-value pairs controlling how the XML
is parsed.
+ """,
examples = """
Examples:
> SELECT _FUNC_('<p><a>1</a><b>0.8</b></p>', 'a INT, b DOUBLE');
@@ -139,6 +146,11 @@ case class XmlToStructs(
*/
@ExpressionDescription(
usage = "_FUNC_(xml[, options]) - Returns schema in the DDL format of XML
string.",
+ arguments = """
+ Arguments:
+ * xml - A foldable string expression containing an XML record whose
schema is inferred.
+ * options - Optional. A map of key-value pairs controlling how the XML
is parsed.
+ """,
examples = """
Examples:
> SELECT _FUNC_('<p><a>1</a></p>');
@@ -223,6 +235,11 @@ case class SchemaOfXml(
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(expr[, options]) - Returns a XML string with a given struct
value",
+ arguments = """
+ Arguments:
+ * expr - A struct-valued expression to convert to an XML string.
+ * options - Optional. A map of key-value pairs controlling how the XML
is generated.
+ """,
examples = """
Examples:
> SELECT _FUNC_(named_struct('a', 1, 'b', 2));
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/expressions/ExpressionInfoSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/expressions/ExpressionInfoSuite.scala
index e38c9e5f20b8..e61299161fdf 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/expressions/ExpressionInfoSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/expressions/ExpressionInfoSuite.scala
@@ -174,11 +174,39 @@ class ExpressionInfoSuite extends SharedSparkSession {
}
test("SPARK-32870: Default expressions in FunctionRegistry should have their
" +
- "usage, examples, since, and group filled") {
+ "usage, examples, arguments, since, and group filled") {
val ignoreSet = Set(
// Cast aliases do not need examples
"org.apache.spark.sql.catalyst.expressions.Cast")
+ // Functions that take no arguments are exempt from the `arguments`
documentation
+ // requirement, since there is nothing to describe.
+ val noArgumentsSet = Set(
+ "org.apache.spark.sql.catalyst.expressions.Collations",
+ "org.apache.spark.sql.catalyst.expressions.CumeDist",
+ "org.apache.spark.sql.catalyst.expressions.CurDateExpressionBuilder",
+ "org.apache.spark.sql.catalyst.expressions.CurrentCatalog",
+ "org.apache.spark.sql.catalyst.expressions.CurrentDatabase",
+ "org.apache.spark.sql.catalyst.expressions.CurrentDate",
+ "org.apache.spark.sql.catalyst.expressions.CurrentPath",
+ "org.apache.spark.sql.catalyst.expressions.CurrentTimeZone",
+ "org.apache.spark.sql.catalyst.expressions.CurrentTimestamp",
+ "org.apache.spark.sql.catalyst.expressions.CurrentUser",
+ "org.apache.spark.sql.catalyst.expressions.EulerNumber",
+ "org.apache.spark.sql.catalyst.expressions.InputFileBlockLength",
+ "org.apache.spark.sql.catalyst.expressions.InputFileBlockStart",
+ "org.apache.spark.sql.catalyst.expressions.InputFileName",
+ "org.apache.spark.sql.catalyst.expressions.LocalTimestamp",
+ "org.apache.spark.sql.catalyst.expressions.MonotonicallyIncreasingID",
+ "org.apache.spark.sql.catalyst.expressions.Now",
+ "org.apache.spark.sql.catalyst.expressions.Pi",
+ "org.apache.spark.sql.catalyst.plans.logical.PythonWorkerLogs",
+ "org.apache.spark.sql.catalyst.expressions.RowNumber",
+ "org.apache.spark.sql.catalyst.expressions.SQLKeywords",
+ "org.apache.spark.sql.catalyst.expressions.SparkPartitionID",
+ "org.apache.spark.sql.catalyst.expressions.SparkVersion",
+ "org.apache.spark.sql.catalyst.expressions.Uuid")
+
spark.sessionState.functionRegistry.listFunction().foreach { funcId =>
val info = spark.sessionState.catalog.lookupFunctionInfo(funcId)
if (!ignoreSet.contains(info.getClassName)) {
@@ -189,6 +217,9 @@ class ExpressionInfoSuite extends SharedSparkSession {
assert(info.getSince.matches("[0-9]+\\.[0-9]+\\.[0-9]+"))
assert(info.getGroup.nonEmpty)
+ if (!noArgumentsSet.contains(info.getClassName)) {
+ assert(info.getArguments.nonEmpty)
+ }
if (info.getArguments.nonEmpty) {
assert(info.getArguments.startsWith("\n Arguments:\n"))
assert(info.getArguments.endsWith("\n "))
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
