Raza Jafri created SPARK-41207:
----------------------------------
Summary: Regression in IntegralDivide
Key: SPARK-41207
URL: https://issues.apache.org/jira/browse/SPARK-41207
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.4.0
Reporter: Raza Jafri
Fix For: 3.4.0
There has been a regression in Integral Divide after the removal of
PromotePrecision from Spark 3.4.0.
{code:java}
scala> val data = Seq(Row(BigDecimal("-7.70892"),
BigDecimal("4.27138661282262736522411173299611831E+40")))
scala> val simpleSchema = StructType(Array(
| StructField("a", DecimalType(6,5),true),
| StructField("b", DecimalType(36,-5), true)))
scala> val df = spark.createDataFrame(spark.sparkContext.parallelize(data),
simpleSchema)
{code}
The above statements result in an AnalysisException thrown
{code:java}
org.apache.spark.sql.AnalysisException: Decimal scale (0) cannot be greater
than precision (-4).
at
org.apache.spark.sql.errors.QueryCompilationErrors$.decimalCannotGreaterThanPrecisionError(QueryCompilationErrors.scala:2237)
at org.apache.spark.sql.types.DecimalType.<init>(DecimalType.scala:49)
at org.apache.spark.sql.types.DecimalType$.bounded(DecimalType.scala:164)
at
org.apache.spark.sql.catalyst.expressions.IntegralDivide.resultDecimalType(arithmetic.scala:868)
at
org.apache.spark.sql.catalyst.expressions.BinaryArithmetic.dataType(arithmetic.scala:238)
at
org.apache.spark.sql.catalyst.expressions.IntegralDivide.org$apache$spark$sql$catalyst$expressions$DivModLike$$super$dataType(arithmetic.scala:842)
{code}
I believe this is happening because we aren't promoting the precision like we
were before this
[PR|https://github.com/apache/spark/commit/301a13963808d1ad44be5cacf0a20f65b853d5a2]
went in. Without promoting precision the resultDecimalType in the example
above tries to return a Decimal with precision of -4 and scale of 0 which is
invalid
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
