This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 0bb7ac687922 [SPARK-60970][SQL] Use `Cast.canUpCast` for the
nullability of `try_cast` from decimal to integral types
0bb7ac687922 is described below
commit 0bb7ac687922e015b123b12d6cd4de5a52ec087b
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Nov 5 20:33:23 2026 -0800
[SPARK-60970][SQL] Use `Cast.canUpCast` for the nullability of `try_cast`
from decimal to integral types
### What changes were proposed in this pull request?
This PR removes the special case in `Cast.nullable` that always makes
`try_cast` from a decimal to an integral type nullable, so that it follows
`Cast.canUpCast` like the other type pairs.
```scala
case (_: StringType, BinaryType) => child.nullable
- // TODO: Follow `Cast.canUpCast` as for the other types. ...
- case (_: DecimalType, _: IntegralType) => true
case _ => child.nullable || !Cast.canUpCast(child.dataType, dataType)
```
Since SPARK-60943 (#60145), `Cast.canUpCast` from a decimal to an integral
type is true exactly when every value fits: the scale is 0 and the precision is
at most 2, 4, 9 and 18 for `TINYINT`, `SMALLINT`, `INT` and `BIGINT`.
`try_cast` of a non-null value of such a type never returns `NULL`, so this PR
makes it non-nullable when its input is not nullable. Every other decimal to
integral `try_cast` stays nullable, including the ones with a positive or
negative scale.
### Why are the changes needed?
SPARK-40054 added the special case with a TODO in Apache Spark
[v3.4.0](https://github.com/apache/spark/releases/tag/v3.4.0) (2023-04-07),
because `Cast.canUpCast` allowed decimal to integral casts that can overflow,
`DECIMAL(10, 0)` to `INT` (fixed by SPARK-40389) and `DECIMAL(19, 0)` to
`BIGINT` (fixed by SPARK-60943). Now that `Cast.canUpCast` is exact, the
special case is only conservative, and it is inconsistent with the other
up-casts:
- `try_cast(i AS BIGINT)` is non-nullable for a non-null `INT` input `i`,
but `try_cast(d AS BIGINT)` is nullable for a non-null `DECIMAL(18, 0)` input
`d`.
- The special case applies only at the top level. A decimal to integral
cast nested in an array or a struct already follows `Cast.canUpCast`, e.g.
`try_cast(array(d) AS ARRAY<BIGINT>)` is non-nullable.
The accurate nullability also lets the optimizer simplify null checks, e.g.
`try_cast(abs(d) AS BIGINT) IS NULL` becomes `false`.
### Does this PR introduce _any_ user-facing change?
Yes. The result of `try_cast` from `DECIMAL(p, 0)` to an integral type is
non-nullable when its input is not nullable and `p` is at most 2, 4, 9 and 18
for `TINYINT`, `SMALLINT`, `INT` and `BIGINT`. This changes the nullability of
such columns in the schema. Query results do not change.
```scala
scala> spark.sql("SELECT try_cast(123456789012345678BD AS BIGINT) AS
c").printSchema()
```
**BEFORE**
```
root
|-- c: long (nullable = true)
```
**AFTER**
```
root
|-- c: long (nullable = false)
```
### How was this patch tested?
Pass the CIs with the newly added test case.
In addition, I compared the nullability of `try_cast` from a non-null
`DECIMAL(p, s)` to each integral type with the evaluation of the largest and
the smallest value of the type, in both the interpreted and the codegen paths,
for every precision `p` and every scale `s` from -3 to `p` (3,572 cases,
negative scales with `spark.sql.legacy.allowNegativeScaleOfDecimal`). 33 cases
become non-nullable, the ones listed above, and none of them returns `NULL`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #60152 from dongjoon-hyun/SPARK-60970.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../scala/org/apache/spark/sql/catalyst/expressions/Cast.scala | 4 ----
.../apache/spark/sql/catalyst/expressions/CastSuiteBase.scala | 9 +++++++++
2 files changed, 9 insertions(+), 4 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
index 4de9d57818ec..a6982c7b3bf4 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
@@ -816,10 +816,6 @@ case class Cast(
} else {
(child.dataType, dataType) match {
case (_: StringType, BinaryType) => child.nullable
- // TODO: Follow `Cast.canUpCast` as for the other types. It no longer
allows a decimal to
- // integral cast that can overflow (SPARK-60943), but following it
here would change
- // the nullability of `try_cast` from a decimal to an integral
type.
- case (_: DecimalType, _: IntegralType) => true
case _ => child.nullable || !Cast.canUpCast(child.dataType, dataType)
}
}
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala
index 297340e6b65f..db6fdf0dbc8f 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala
@@ -949,6 +949,15 @@ abstract class CastSuiteBase extends SparkFunSuite with
ExpressionEvalHelper {
}
}
+ test("SPARK-60970: try-cast nullability follows up-cast admissibility from
decimal to integral") {
+ def tryCast(from: DataType, to: DataType): Cast =
+ Cast(AttributeReference("c", from, nullable = false)(), to, evalMode =
EvalMode.TRY)
+ // Every DECIMAL(18, 0) value fits in a long, so a non-null child stays
non-null.
+ assert(!tryCast(DecimalType(18, 0), LongType).nullable)
+ // A DECIMAL(19, 0) value can overflow a long, so the try-cast is nullable.
+ assert(tryCast(DecimalType(19, 0), LongType).nullable)
+ }
+
test("SPARK-40389: canUpCast: return false if casting decimal to integral
types can cause" +
" overflow") {
Seq(ByteType, ShortType, IntegerType, LongType).foreach { integralType =>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]