mgaido91 commented on a change in pull request #20350: [SPARK-23179][SQL]
Support option to throw exception if overflow occurs
URL: https://github.com/apache/spark/pull/20350#discussion_r296442641
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -1441,6 +1441,16 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val DECIMAL_OPERATIONS_NULL_ON_OVERFLOW =
+ buildConf("spark.sql.decimalOperations.nullOnOverflow")
Review comment:
> A DB does not have to follow the SQL standard completely in every corners.
The current behavior in Spark is by design and I don't think that's nonsense.
I am sorry, but I don't really agree with you on this. I see the discussion
is a bit OT, but I'd like just to explain the reasons of my opinion. SQL is a
declarative language and here we are coupling the result/behavior to the
specific execution language we are using. Spark is cross-language, but for
arithmetic operations overflow works in a very peculiar way of the language we
use which is:
- against SQL standards and no other DB works differently from SQL
standards w.r.t. this, so very surprising (at least) for SQL users;
- different from what happens in Python and in R when you overflow in those
languages (an Int becomes long and so on there);
So there in no Spark user other than Scala/Java ones who might understand
the behavior Spark has in those cases. Sorry for being a bit OT, anyway.
> My question is if we need one config for overflow, or 2 configs for
decimal and non-decimal.
Yes, this is the main point here. IMHO, I'd prefer 2 configs because when
the config is turned off, the behavior is completely different: in once case it
returns null, in the other we return wrong results. But I see also the value in
reducing as much as possible the number of configs, which is already pretty
big. So I'd prefer 2 configs, but if you and the community thinks 1 it is
better, I can update the PR in order to make this config more generic.
Thanks for your feedbacks and the discussion!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
