mbutrovich opened a new issue, #7659:
URL: https://github.com/apache/datafusion-comet/issues/7659

   ### What is the problem the feature request solves?
   
   
[`DecimalPrecision.promote`](https://github.com/apache/datafusion-comet/blob/36ab96ad8a692ddd0cc0bd89bd645d74a11b1c6f/spark/src/main/scala/org/apache/spark/sql/comet/DecimalPrecision.scala#L74-L84)
 wraps every decimal `Add`, `Subtract`, and `Multiply` in `CheckOverflow`. When 
the operation takes the narrow path in the planner and runs as a DataFusion 
[`BinaryExpr`](https://github.com/apache/datafusion-comet/blob/36ab96ad8a692ddd0cc0bd89bd645d74a11b1c6f/native/core/src/execution/planner.rs#L1588),
 no inputs that fit their declared types can produce a result that exceeds the 
result precision.
   
   - For `+` and `-`, the planner takes the narrow path when 
[`decimal128_add_sub_width`](https://github.com/apache/datafusion-comet/blob/36ab96ad8a692ddd0cc0bd89bd645d74a11b1c6f/native/spark-expr/src/math_funcs/modulo_expr.rs#L137-L149),
 `max(s1, s2) + max(p1 - s1, p2 - s2)`, is below 38 
([condition](https://github.com/apache/datafusion-comet/blob/36ab96ad8a692ddd0cc0bd89bd645d74a11b1c6f/native/core/src/execution/planner.rs#L1446-L1453)).
 Spark's result type is then [`(max(s1, s2) + max(p1 - s1, p2 - s2) + 1, 
max(s1, 
s2))`](https://github.com/apache/spark/blob/cecbd5d846a2cdd891a4d9f0bda302b58b02092a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala#L443-L451)
 unchanged, because 
[`adjustPrecisionScale`](https://github.com/apache/spark/blob/cecbd5d846a2cdd891a4d9f0bda302b58b02092a/sql/api/src/main/scala/org/apache/spark/sql/types/DecimalType.scala#L180-L182)
 and 
[`bounded`](https://github.com/apache/spark/blob/cecbd5d846a2cdd891a4d9f0bda302b58b020
 
92a/sql/api/src/main/scala/org/apache/spark/sql/types/DecimalType.scala#L144-L146)
 adjust nothing at or below 38 digits. Each operand rescaled to `max(s1, s2)` 
is below `10^(max(p1 - s1, p2 - s2) + max(s1, s2))` in absolute value, so their 
sum or difference is below `10^(result precision)`.
   - For `*`, the planner takes the narrow path when `p1 + p2` is below 38. 
Spark's result type is then [`(p1 + p2 + 1, s1 + 
s2)`](https://github.com/apache/spark/blob/cecbd5d846a2cdd891a4d9f0bda302b58b02092a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala#L630-L638),
 and the product of the unscaled values is below `10^(p1 + p2)`.
   
   So on this path, `CheckOverflow` can only null out or raise for an input 
whose value already exceeds its own declared precision, but it still scans 
every batch.
   
   In TPC-H Q1, every `CheckOverflow` is on this path. Spark types the literal 
`1` as `decimal(1,0)` by default 
([coercion](https://github.com/apache/spark/blob/cecbd5d846a2cdd891a4d9f0bda302b58b02092a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DecimalPrecisionTypeCoercion.scala#L170-L174),
 
[`spark.sql.legacy.literal.pickMinimumPrecision`](https://github.com/apache/spark/blob/cecbd5d846a2cdd891a4d9f0bda302b58b02092a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L6023-L6031)).
 With `decimal(15,2)` columns, `1 - l_discount` and `1 + l_tax` are 
`decimal(1,0)` with `decimal(15,2)`, and `l_extendedprice * (1 - l_discount)` 
is `decimal(15,2) * decimal(16,2)`. The outer `* (1 + l_tax)` needs 48 digits, 
so it runs as `WideDecimalBinaryExpr`, where the planner [already drops 
`CheckOverflow`](https://github.com/apache/datafusion-comet/blob/36ab96ad8a692ddd0cc0bd89bd645d74a11b1c6f/native/core/src/execution/planner.rs#L858-L875).
 #7658 measured t
 he `CheckOverflow` scan at about 5% of native CPU samples in Q1 at SF10.
   
   ### Describe the potential solution
   
   When the child of `CheckOverflow` is a `Decimal128` `+`, `-`, or `*` on the 
narrow path and its type equals the `CheckOverflow` target type, the planner 
could return the child without the wrapper, the way it already does for 
`WideDecimalBinaryExpr`. The same type check could instead run in 
`DecimalPrecision.promote`, so the wrapper is never serialized.
   
   ### Additional context
   
   The open question is compatibility for inputs that violate their own 
declared type. Spark's `Add`, `Subtract`, and `Multiply` call 
[`checkDecimalOverflow`](https://github.com/apache/spark/blob/cecbd5d846a2cdd891a4d9f0bda302b58b02092a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala#L457-L459)
 on every result, so Spark nulls out or raises on such a value, and Comet would 
pass it through. One way to get such a value is a Parquet file whose unscaled 
values exceed the column's declared precision, if the reader doesn't reject it. 
I haven't checked whether Spark's or Comet's Parquet readers validate this. 
Does Comet need to match Spark on this input, or can it be documented as a 
difference for malformed data?
   
   ### Willingness to contribute
   
   I cannot contribute a fix for this bug at this time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to