This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 0bb7ac687922 [SPARK-60970][SQL] Use `Cast.canUpCast` for the 
nullability of `try_cast` from decimal to integral types
0bb7ac687922 is described below

commit 0bb7ac687922e015b123b12d6cd4de5a52ec087b
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Nov 5 20:33:23 2026 -0800

    [SPARK-60970][SQL] Use `Cast.canUpCast` for the nullability of `try_cast` 
from decimal to integral types
    
    ### What changes were proposed in this pull request?
    
    This PR removes the special case in `Cast.nullable` that always makes 
`try_cast` from a decimal to an integral type nullable, so that it follows 
`Cast.canUpCast` like the other type pairs.
    
    ```scala
           case (_: StringType, BinaryType) => child.nullable
    -      // TODO: Follow `Cast.canUpCast` as for the other types. ...
    -      case (_: DecimalType, _: IntegralType) => true
           case _ => child.nullable || !Cast.canUpCast(child.dataType, dataType)
    ```
    
    Since SPARK-60943 (#60145), `Cast.canUpCast` from a decimal to an integral 
type is true exactly when every value fits: the scale is 0 and the precision is 
at most 2, 4, 9 and 18 for `TINYINT`, `SMALLINT`, `INT` and `BIGINT`. 
`try_cast` of a non-null value of such a type never returns `NULL`, so this PR 
makes it non-nullable when its input is not nullable. Every other decimal to 
integral `try_cast` stays nullable, including the ones with a positive or 
negative scale.
    
    ### Why are the changes needed?
    
    SPARK-40054 added the special case with a TODO in Apache Spark 
[v3.4.0](https://github.com/apache/spark/releases/tag/v3.4.0) (2023-04-07), 
because `Cast.canUpCast` allowed decimal to integral casts that can overflow, 
`DECIMAL(10, 0)` to `INT` (fixed by SPARK-40389) and `DECIMAL(19, 0)` to 
`BIGINT` (fixed by SPARK-60943). Now that `Cast.canUpCast` is exact, the 
special case is only conservative, and it is inconsistent with the other 
up-casts:
    
    - `try_cast(i AS BIGINT)` is non-nullable for a non-null `INT` input `i`, 
but `try_cast(d AS BIGINT)` is nullable for a non-null `DECIMAL(18, 0)` input 
`d`.
    - The special case applies only at the top level. A decimal to integral 
cast nested in an array or a struct already follows `Cast.canUpCast`, e.g. 
`try_cast(array(d) AS ARRAY<BIGINT>)` is non-nullable.
    
    The accurate nullability also lets the optimizer simplify null checks, e.g. 
`try_cast(abs(d) AS BIGINT) IS NULL` becomes `false`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. The result of `try_cast` from `DECIMAL(p, 0)` to an integral type is 
non-nullable when its input is not nullable and `p` is at most 2, 4, 9 and 18 
for `TINYINT`, `SMALLINT`, `INT` and `BIGINT`. This changes the nullability of 
such columns in the schema. Query results do not change.
    
    ```scala
    scala> spark.sql("SELECT try_cast(123456789012345678BD AS BIGINT) AS 
c").printSchema()
    ```
    
    **BEFORE**
    
    ```
    root
     |-- c: long (nullable = true)
    ```
    
    **AFTER**
    
    ```
    root
     |-- c: long (nullable = false)
    ```
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test case.
    
    In addition, I compared the nullability of `try_cast` from a non-null 
`DECIMAL(p, s)` to each integral type with the evaluation of the largest and 
the smallest value of the type, in both the interpreted and the codegen paths, 
for every precision `p` and every scale `s` from -3 to `p` (3,572 cases, 
negative scales with `spark.sql.legacy.allowNegativeScaleOfDecimal`). 33 cases 
become non-nullable, the ones listed above, and none of them returns `NULL`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #60152 from dongjoon-hyun/SPARK-60970.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../scala/org/apache/spark/sql/catalyst/expressions/Cast.scala   | 4 ----
 .../apache/spark/sql/catalyst/expressions/CastSuiteBase.scala    | 9 +++++++++
 2 files changed, 9 insertions(+), 4 deletions(-)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
index 4de9d57818ec..a6982c7b3bf4 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
@@ -816,10 +816,6 @@ case class Cast(
   } else {
     (child.dataType, dataType) match {
       case (_: StringType, BinaryType) => child.nullable
-      // TODO: Follow `Cast.canUpCast` as for the other types. It no longer 
allows a decimal to
-      //       integral cast that can overflow (SPARK-60943), but following it 
here would change
-      //       the nullability of `try_cast` from a decimal to an integral 
type.
-      case (_: DecimalType, _: IntegralType) => true
       case _ => child.nullable || !Cast.canUpCast(child.dataType, dataType)
     }
   }
diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala
 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala
index 297340e6b65f..db6fdf0dbc8f 100644
--- 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala
+++ 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala
@@ -949,6 +949,15 @@ abstract class CastSuiteBase extends SparkFunSuite with 
ExpressionEvalHelper {
     }
   }
 
+  test("SPARK-60970: try-cast nullability follows up-cast admissibility from 
decimal to integral") {
+    def tryCast(from: DataType, to: DataType): Cast =
+      Cast(AttributeReference("c", from, nullable = false)(), to, evalMode = 
EvalMode.TRY)
+    // Every DECIMAL(18, 0) value fits in a long, so a non-null child stays 
non-null.
+    assert(!tryCast(DecimalType(18, 0), LongType).nullable)
+    // A DECIMAL(19, 0) value can overflow a long, so the try-cast is nullable.
+    assert(tryCast(DecimalType(19, 0), LongType).nullable)
+  }
+
   test("SPARK-40389: canUpCast: return false if casting decimal to integral 
types can cause" +
     " overflow") {
     Seq(ByteType, ShortType, IntegerType, LongType).foreach { integralType =>


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to