mgaido91 commented on a change in pull request #20350: [SPARK-23179][SQL]
Support option to throw exception if overflow occurs
URL: https://github.com/apache/spark/pull/20350#discussion_r296440219
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -1441,6 +1441,16 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val DECIMAL_OPERATIONS_NULL_ON_OVERFLOW =
+ buildConf("spark.sql.decimalOperations.nullOnOverflow")
Review comment:
Thanks for taking a look at this @cloud-fan !
Yes, that case (non-decimal) is handled in
https://github.com/apache/spark/pull/21599. I'd say that, in the non-decimal
case, the situation is pretty different. Indeed, overflow in decimal operation
is handled by Spark now, converting overflow operations to `null`; while
overflow in operation on non-decimal isn't handled at all currently.
In non-decimal operations, indeed we return a wrong value (the java way). So
IMHO, the non-decimal case current behavior doesn't make any sense at all
(considering this is SQL and not a low level language like Java/Scala) and
keeping its current behavior makes no sense (we already discussed this in that
PR actually).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
