cloud-fan commented on issue #27627: [WIP][SPARK-28067][SQL] Fix incorrect results for decimal aggregate sum by returning null on decimal overflow URL: https://github.com/apache/spark/pull/27627#issuecomment-596918804 @skambha thanks for your investigation! Very helpful. I'm trying to determine the scope of this bug. Your analysis is for global aggregate (no GROUP BY), when we only write out the sum value at the end of partial aggregate. I tried with group by ``` df.select(expr(s"cast('$decimalStr' as decimal (38, 18)) as d"), lit(1).as("key")) .groupBy("key").agg(sum($"d")).show java.lang.ArithmeticException: Decimal precision 39 exceeds max precision 38 at org.apache.spark.sql.types.Decimal.set(Decimal.scala:122) at org.apache.spark.sql.types.Decimal$.apply(Decimal.scala:574) at org.apache.spark.sql.types.Decimal.apply(Decimal.scala) at org.apache.spark.sql.catalyst.expressions.UnsafeRow.getDecimal(UnsafeRow.java:393) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.agg_doConsume_0$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.agg_doAggregateWithKeys_1$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.agg_doAggregateWithKeys_0$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) ... ``` Do you what happens there? I'd expect we write overflowed decimal as null to the aggregate hash map, but it seems not the case.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
