wangyum opened a new pull request #25137: [SPARK-28348][SQL] Decimal precision promotion for binary arithmetic with casted decimal type URL: https://github.com/apache/spark/pull/25137 ## What changes were proposed in this pull request? How to reproduce this issue: ```sql create table spark_28348 using parquet as select * from values (cast(-34338492.215397047 as decimal(38, 18))) as v(c1); ``` ```shell scala> spark.sql("select cast(c1 * cast(-34338492.215397047 as decimal(38, 18)) as decimal(38, 18)) as c1 from spark_28348").show(false) +-----------------------------------+ |c1 | +-----------------------------------+ |1179132047626883.596862000000000000| +-----------------------------------+ scala> spark.sql("select cast(c1 * cast(-34338492.215397047 as decimal(38, 18)) as decimal(38, 18)) as c1 from spark_28348").explain(false); == Physical Plan == *(1) Project [cast(CheckOverflow((promote_precision(c1#1) * -34338492.215397047000000000), DecimalType(38,6), true) as decimal(38,18)) AS c1#13] +- *(1) ColumnarToRow +- FileScan parquet default.spark_28348[c1#1] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/user/hive/warehouse/spark_28348], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<c1:decimal(38,18)> ``` It first `CheckOverflow` with `DecimalType(38,6)` and then cast to `decimal(38,18)`. We can use `decimal(38,18)` to `CheckOverflow`. This has two benefits: 1. Reduce one cast. 2. Higher data precision. After this PR: ```shell scala> spark.sql("select cast(c1 * cast(-34338492.215397047 as decimal(38, 18)) as decimal(38, 18)) as c1 from spark_28348").show(false) +-----------------------------------+ |c1 | +-----------------------------------+ |1179132047626883.596862135856320209| +-----------------------------------+ scala> spark.sql("select cast(c1 * cast(-34338492.215397047 as decimal(38, 18)) as decimal(38, 18)) as c1 from spark_28348").explain(false); == Physical Plan == *(1) Project [CheckOverflow((promote_precision(c1#1) * -34338492.215397047000000000), DecimalType(38,18), true) AS c1#17] +- *(1) ColumnarToRow +- FileScan parquet default.spark_28348[c1#1] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/user/hive/warehouse/spark_28348], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<c1:decimal(38,18)> ``` ## How was this patch tested? unit tests
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
