JoshRosen commented on issue #20350: [SPARK-23179][SQL] Support option to throw exception if overflow occurs during Decimal arithmetic URL: https://github.com/apache/spark/pull/20350#issuecomment-506391333 > Actually this is a weird operator, because it does throw an exception on overflow in interpreted mode, while in codegen it doesn't. Moreover, it throw a IllegalArgumentException, instead of the ArithmeticException Since this is a weird behavior of the operator itself, I'd prefer having a PR targeting that specific operator and addressing its behavior totally, since it need to be revisited carefully. Are there concerns on this plan? +1; it sounds like the pre-existing difference between the codegen and interpreted paths is a separate, pre-existing bug. It's also especially hard to reason about because (AFAIK) the paths in `DecimalAggregates` are only used for certain sizes of decimals, so the behavioral inconsistency can be triggered by a combination of precision/scale and codegen/interpreted (which is really confusing!). For consistency, I think we should: 1. Ensure that the `agg(sum)` codegen and interpreted paths behave the same w.r.t `nullOnOverflow == true` (the default / 2.x behavior). 2. Respect the `nullOnOverflow` flag in `agg(sum)` codegen. Let's make a followup JIRA for this change and a separate JIRA for the encoder changes @mickjermsurawong-stripe discussed in his comment (I can loop back later this morning or afternoon to help file these).
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
