revans2 commented on a change in pull request #35913:
URL: https://github.com/apache/spark/pull/35913#discussion_r831095223
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -1783,7 +1783,9 @@ object functions {
* @group math_funcs
* @since 1.4.0
*/
- def ceil(e: Column): Column = ceil(e, lit(0))
+ def ceil(e: Column): Column = withExpr {
+ UnresolvedFunction(Seq("ceil"), Seq(e.expr), isDistinct = false)
Review comment:
It is a bug.
```scala
scala> spark.range(1).selectExpr("id", "ceil(id) as one_arg_sql", "ceil(id,
0) as two_arg_sql").select(col("*"), ceil(col("id")).alias("one_arg_func"),
ceil(col("id"), lit(0)).alias("two_arg_func")).printSchema
root
|-- id: long (nullable = false)
|-- one_arg_sql: long (nullable = true)
|-- two_arg_sql: decimal(20,0) (nullable = true)
|-- one_arg_func: decimal(20,0) (nullable = true)
|-- two_arg_func: decimal(20,0) (nullable = true)
scala> spark.range(1).selectExpr("cast(id as double) as
id").selectExpr("id", "ceil(id) as one_arg_sql", "ceil(id, 0) as
two_arg_sql").select(col("*"), ceil(col("id")).alias("one_arg_func"),
ceil(col("id"), lit(0)).alias("two_arg_func")).printSchema
root
|-- id: double (nullable = false)
|-- one_arg_sql: long (nullable = true)
|-- two_arg_sql: decimal(30,0) (nullable = true)
|-- one_arg_func: decimal(30,0) (nullable = true)
|-- two_arg_func: decimal(30,0) (nullable = true)
```
Without this patch the SQL and scala APIs produce different results and the
scala API produces a result that is different from what was in Spark 3.2.
I documented this in the JIRA
https://issues.apache.org/jira/browse/SPARK-38604
After this patch the single argument version behaves like it did in 3.2 and
is also consistent with the SQL API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
