This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 939517012a19 [SPARK-59373][SQL][PYTHON][DOC] Fix the documented return
type of ceil and floor with a scale argument
939517012a19 is described below
commit 939517012a19867d634d48302b1425627dfd9bfc
Author: Varun Bhandary <[email protected]>
AuthorDate: Tue Sep 29 18:24:23 2026 +0900
[SPARK-59373][SQL][PYTHON][DOC] Fix the documented return type of ceil and
floor with a scale argument
### What changes were proposed in this pull request?
`floor`'s Python docstring states a return type that is wrong whenever
`scale` is passed, and
none of the Python, Scala or SQL documentation says when these functions
return a long and when
they return a decimal. This removes the wrong line and states the rule on
all three surfaces.
- `python/pyspark/sql/functions/builtin.py`: the `Returns` block for
`floor`, `ceil` and `ceiling`.
- `sql/api/src/main/scala/org/apache/spark/sql/functions.scala`: eight
scaladoc sites. The three
two argument overloads now say they return a decimal, rather than
offering a choice of two
types, and the one argument overloads say a long, or a decimal when the
input is a decimal.
-
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala`:
a `note` on the `ceil` and `floor` `ExpressionDescription`, since the SQL
function reference
said nothing about the return type at all.
Documentation only, no behaviour change.
One thing I left alone deliberately. The `note` on these functions says
they are affected by
`spark.sql.ansi.enabled`. `RoundCeil` and `RoundFloor` do not override
`RoundBase.ansiEnabled`,
which defaults to `false`, while `Round`, `BRound` and `Truncate` all do,
so the config looks
like it applies only to the one argument forms. I could not confirm that by
running anything, so
I have not touched it. Happy to fix it here or in a follow up if someone
can confirm.
### Why are the changes needed?
The `Returns` block in `pyspark.sql.functions.floor` contradicts itself:
```
nearest integer that is less than or equal to given value.
Returns a column that evaluates to a long or decimal.
```
The first line is a leftover. SPARK-45131 added the second one to `ceil`,
`ceiling` and `floor`,
and it replaced the vague wording in `ceil` and `ceiling`, but left the
wrong line standing in
`floor`. The docstring's own example disproves it, since `floor(2.1267, 2)`
returns
`Decimal('2.12')`, which is not an integer.
"a long or decimal" is accurate but never says which case gives which, and
the rule is not the
one a reader would guess, because passing `scale` returns a decimal for
every input type,
integers included:
```
floor(double) is bigint, floor(double, 1) is decimal(17,1)
ceil(double) is bigint, ceil(double, 1) is decimal(17,1)
floor(float, 1) is decimal(9,1)
floor(bigint, 1) is decimal(21,0)
floor(int, 1) is decimal(11,0)
floor(decimal(10,3), 1) is decimal(9,1)
```
So passing `scale` produces a decimal for every input type, including
integral ones, and nothing
said so anywhere.
An earlier revision of this PR also claimed that `round`, `bround` and
`truncate` return the same
type as `expr`, and drew that as a contrast. That is wrong and the claim
has been removed. Those
functions preserve a *primitive* numeric input's type, but for a decimal
input the precision and
scale still change: on 4.0.0, `round(decimal(10,3), 1)` returns
`decimal(9,1)` and
`round(decimal(10,0))` returns `decimal(11,0)`. String input is coerced to
double. Documenting
`ceil` and `floor` accurately does not require the comparison, so the notes
now state only their
own return type.
This documents current behaviour rather than proposing a change to it.
SPARK-38604 records that
these return types were discussed on #34729, when SPARK-37475 added the
scale parameter, and
chosen deliberately: keep the old behaviour when no scale is given, use the
new one when a scale
is passed.
### Does this PR introduce _any_ user-facing change?
No. Documentation only, which `.github/PULL_REQUEST_TEMPLATE` states is not
considered a
user-facing change. The Python docstrings, the Scala scaladoc and the SQL
function reference for
`ceil`, `ceiling` and `floor` now state the return type; no behaviour
changes.
### How was this patch tested?
The types in the table above were measured on 4.0.0 rather than read off
the source, by taking
`spark.sql("SELECT <expr>").schema` for each expression.
`ruff check` and `ruff format --check` both pass on
`python/pyspark/sql/functions/builtin.py`,
using the 0.14.8 pinned in `dev/requirements.txt`. The two new `note`
strings satisfy
`ExpressionInfo`'s validation, which requires a note to contain four
consecutive spaces and to
end in two. No `examples` block changed, so `sql-expression-schema.md` is
unaffected.
I did not build Spark locally, so the Scala compile, scalastyle and the
PySpark doctests are
covered by CI here.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code 2.1.266 (Claude Opus 5)
Closes #58662 from vb-dbrks/SPARK-59373-ceil-floor-return-type.
Authored-by: Varun Bhandary <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
python/pyspark/sql/functions/builtin.py | 15 +++++++++------
.../src/main/scala/org/apache/spark/sql/functions.scala | 16 ++++++++--------
.../spark/sql/catalyst/expressions/mathExpressions.scala | 4 ++++
3 files changed, 21 insertions(+), 14 deletions(-)
diff --git a/python/pyspark/sql/functions/builtin.py
b/python/pyspark/sql/functions/builtin.py
index a25d3400c479..e7c3016a002d 100644
--- a/python/pyspark/sql/functions/builtin.py
+++ b/python/pyspark/sql/functions/builtin.py
@@ -2787,8 +2787,9 @@ def ceil(col: "ColumnOrName", scale:
Optional[Union[Column, int]] = None) -> Col
Returns
-------
:class:`~pyspark.sql.Column`
- A column for the computed results.
- Returns a column that evaluates to a long or decimal.
+ The smallest number not less than ``col``. Without ``scale`` the
result is a
+ long, or a decimal when ``col`` is a decimal. When ``scale`` is given
the result
+ is always a decimal, whatever the type of ``col``.
See Also
--------
@@ -2854,8 +2855,9 @@ def ceiling(col: "ColumnOrName", scale:
Optional[Union[Column, int]] = None) ->
Returns
-------
:class:`~pyspark.sql.Column`
- A column for the computed results.
- Returns a column that evaluates to a long or decimal.
+ The smallest number not less than ``col``. Without ``scale`` the
result is a
+ long, or a decimal when ``col`` is a decimal. When ``scale`` is given
the result
+ is always a decimal, whatever the type of ``col``.
See Also
--------
@@ -3283,8 +3285,9 @@ def floor(col: "ColumnOrName", scale:
Optional[Union[Column, int]] = None) -> Co
Returns
-------
:class:`~pyspark.sql.Column`
- nearest integer that is less than or equal to given value.
- Returns a column that evaluates to a long or decimal.
+ The largest number not greater than ``col``. Without ``scale`` the
result is a
+ long, or a decimal when ``col`` is a decimal. When ``scale`` is given
the result
+ is always a decimal, whatever the type of ``col``.
Notes
-----
diff --git a/sql/api/src/main/scala/org/apache/spark/sql/functions.scala
b/sql/api/src/main/scala/org/apache/spark/sql/functions.scala
index cfbc73c20163..9160e618cda0 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/functions.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/functions.scala
@@ -5292,7 +5292,7 @@ object functions {
* @group math_funcs
* @since 3.3.0
* @return
- * Returns a column that evaluates to a long or decimal.
+ * Returns a column that evaluates to a decimal, whatever the type of `e`.
*
* @note
* Affected by these public SQL configurations:
@@ -5308,7 +5308,7 @@ object functions {
* @group math_funcs
* @since 1.4.0
* @return
- * Returns a column that evaluates to a long or decimal.
+ * Returns a column that evaluates to a long, or a decimal when `e` is a
decimal.
*
* @note
* Affected by these public SQL configurations:
@@ -5324,7 +5324,7 @@ object functions {
* @group math_funcs
* @since 1.4.0
* @return
- * Returns a column that evaluates to a long or decimal.
+ * Returns a column that evaluates to a long, or a decimal when the column
is a decimal.
*
* @note
* Affected by these public SQL configurations:
@@ -5343,7 +5343,7 @@ object functions {
* @group math_funcs
* @since 3.5.0
* @return
- * Returns a column that evaluates to a long or decimal.
+ * Returns a column that evaluates to a decimal, whatever the type of `e`.
*
* @note
* Affected by these public SQL configurations:
@@ -5359,7 +5359,7 @@ object functions {
* @group math_funcs
* @since 3.5.0
* @return
- * Returns a column that evaluates to a long or decimal.
+ * Returns a column that evaluates to a long, or a decimal when `e` is a
decimal.
*
* @note
* Affected by these public SQL configurations:
@@ -5537,7 +5537,7 @@ object functions {
* @group math_funcs
* @since 3.3.0
* @return
- * Returns a column that evaluates to a long or decimal.
+ * Returns a column that evaluates to a decimal, whatever the type of `e`.
*
* @note
* Affected by these public SQL configurations:
@@ -5553,7 +5553,7 @@ object functions {
* @group math_funcs
* @since 1.4.0
* @return
- * Returns a column that evaluates to a long or decimal.
+ * Returns a column that evaluates to a long, or a decimal when `e` is a
decimal.
*
* @note
* Affected by these public SQL configurations:
@@ -5569,7 +5569,7 @@ object functions {
* @group math_funcs
* @since 1.4.0
* @return
- * Returns a column that evaluates to a long or decimal.
+ * Returns a column that evaluates to a long, or a decimal when the column
is a decimal.
*
* @note
* Affected by these public SQL configurations:
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
index 0aebdc022314..08da25e1d9c3 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
@@ -345,6 +345,8 @@ trait CeilFloorExpressionBuilderBase extends
ExpressionBuilder {
An expression that evaluates to a numeric.
* scale - The number of decimal places to round to.
An expression that evaluates to an integer. Must be a constant.
+ When `scale` is specified the result is a decimal, whatever the type
of `expr`.
+ When it is omitted the result is a long, or a decimal if `expr` is a
decimal.
""",
examples = """
Examples:
@@ -646,6 +648,8 @@ case class Floor(child: Expression, failOnError: Boolean =
SQLConf.get.ansiEnabl
An expression that evaluates to a numeric.
* scale - The number of decimal places to round to.
An expression that evaluates to an integer. Must be a constant.
+ When `scale` is specified the result is a decimal, whatever the type
of `expr`.
+ When it is omitted the result is a long, or a decimal if `expr` is a
decimal.
""",
examples = """
Examples:
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
