[
https://issues.apache.org/jira/browse/SPARK-59373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Bhandary updated SPARK-59373:
-----------------------------------
Summary: Fix the documented return type of ceil and floor with a scale
argument (was: Fix the documented return type of ceil and floor with a scale
argument)
> Fix the documented return type of ceil and floor with a scale argument
> ----------------------------------------------------------------------
>
> Key: SPARK-59373
> URL: https://issues.apache.org/jira/browse/SPARK-59373
> Project: Spark
> Issue Type: Documentation
> Components: Documentation
> Affects Versions: 5.0.0, 4.4.0
> Reporter: Varun Bhandary
> Priority: Minor
>
> The Returns block in pyspark.sql.functions.floor contradicts itself. It says
> "nearest integer that is less than or equal to given value" and then, on the
> very
> next line, "Returns a column that evaluates to a long or decimal".
> The first line is a leftover from before SPARK-45131 and is wrong whenever
> scale
> is passed. The docstring's own example disproves it: floor(2.1267, 2) returns
> Decimal('2.12'), which is not an integer.
> The second line is accurate but does not say which case gives which, and the
> rule
> is not obvious, because passing scale returns a decimal for every input type,
> integers included. Measured on 4.0.0:
> floor(double) is bigint, and floor(double, 1) is decimal(17,1)
> ceil(double) is bigint, and ceil(double, 1) is decimal(17,1)
> round(double, 1) is double
> floor(bigint, 1) is decimal(21,0)
> So ceil and floor are the only scale taking math functions that do not return
> the
> input's type, and nothing in the documentation says so. round, bround and
> truncate all preserve it.
> This is a documentation fix rather than a behaviour change. SPARK-38604
> records
> that these return types were chosen deliberately when SPARK-37475 added the
> scale parameter.
> The fix is to remove the wrong line, state the rule in the Python and Scala
> docs
> for floor, ceil and ceiling, and add it to the ceil and floor SQL function
> descriptions, which currently say nothing about the return type at all.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
