vb-dbrks opened a new pull request, #58662:
URL: https://github.com/apache/spark/pull/58662
### What changes were proposed in this pull request?
`floor`'s Python docstring states a return type that is wrong whenever
`scale` is passed, and
none of the Python, Scala or SQL documentation says when these functions
return a long and when
they return a decimal. This removes the wrong line and states the rule on
all three surfaces.
- `python/pyspark/sql/functions/builtin.py`: the `Returns` block for
`floor`, `ceil` and `ceiling`.
- `sql/api/src/main/scala/org/apache/spark/sql/functions.scala`: eight
scaladoc sites. The three
two argument overloads now say they return a decimal, rather than offering
a choice of two
types, and the one argument overloads say a long, or a decimal when the
input is a decimal.
-
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala`:
a `note` on the `ceil` and `floor` `@ExpressionDescription`, since the SQL
function reference
said nothing about the return type at all. That is also the one place
these functions sit next
to each other, so it is where the contrast with `round` is worth drawing.
Documentation only, no behaviour change.
One thing I left alone deliberately. The `@note` on these functions says
they are affected by
`spark.sql.ansi.enabled`. `RoundCeil` and `RoundFloor` do not override
`RoundBase.ansiEnabled`,
which defaults to `false`, while `Round`, `BRound` and `Truncate` all do, so
the config looks
like it applies only to the one argument forms. I could not confirm that by
running anything, so
I have not touched it. Happy to fix it here or in a follow up if someone can
confirm.
### Why are the changes needed?
The `Returns` block in `pyspark.sql.functions.floor` contradicts itself:
```
nearest integer that is less than or equal to given value.
Returns a column that evaluates to a long or decimal.
```
The first line is a leftover. SPARK-45131 added the second one to `ceil`,
`ceiling` and `floor`,
and it replaced the vague wording in `ceil` and `ceiling`, but left the
wrong line standing in
`floor`. The docstring's own example disproves it, since `floor(2.1267, 2)`
returns
`Decimal('2.12')`, which is not an integer.
"a long or decimal" is accurate but never says which case gives which, and
the rule is not the
one a reader would guess, because passing `scale` returns a decimal for
every input type,
integers included:
```
floor(double) is bigint, and floor(double, 1) is decimal(17,1)
ceil(double) is bigint, and ceil(double, 1) is decimal(17,1)
round(double, 1) is double
floor(bigint, 1) is decimal(21,0)
```
So `ceil` and `floor` are the only scale taking math functions that do not
return the input's
type. `round`, `bround` and `truncate` all preserve it, and nothing said so
anywhere.
This documents current behaviour rather than proposing a change to it.
SPARK-38604 records that
these return types were discussed on #34729, when SPARK-37475 added the
scale parameter, and
chosen deliberately: keep the old behaviour when no scale is given, use the
new one when a scale
is passed.
### Does this PR introduce _any_ user-facing change?
Yes, documentation only. The Python docstrings, the Scala scaladoc and the
SQL function reference
for `ceil`, `ceiling` and `floor` now state the return type. No behaviour
changes.
### How was this patch tested?
The types in the table above were measured on 4.0.0 rather than read off the
source, by taking
`spark.sql("SELECT <expr>").schema` for each expression.
`ruff check` and `ruff format --check` both pass on
`python/pyspark/sql/functions/builtin.py`,
using the 0.14.8 pinned in `dev/requirements.txt`. The two new `note`
strings satisfy
`ExpressionInfo`'s validation, which requires a note to contain four
consecutive spaces and to
end in two. No `examples` block changed, so `sql-expression-schema.md` is
unaffected.
I did not build Spark locally, so the Scala compile, scalastyle and the
PySpark doctests are
covered by CI here.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code 2.1.266 (Claude Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
